Note: This is test shard 1 of 8.
[==========] Running 8 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from AdminCliTest
[ RUN ] AdminCliTest.RebalancerReportOnly
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240503 13:55:38.410873 6537 test_util.cc:268] Using random seed: 320319332
I20240503 13:55:38.417479 6537 ts_itest-base.cc:112] Starting cluster with:
I20240503 13:55:38.417539 6537 ts_itest-base.cc:113] --------------
I20240503 13:55:38.417557 6537 ts_itest-base.cc:114] 5 tablet servers
I20240503 13:55:38.417572 6537 ts_itest-base.cc:115] 3 replicas per TS
I20240503 13:55:38.417584 6537 ts_itest-base.cc:116] --------------
2024-05-03T13:55:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T13:55:38Z Disabled control of system clock
I20240503 13:55:38.426736 6537 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
/tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.98.126:39627
--webserver_interface=127.6.98.126
--webserver_port=0
--builtin_ntp_servers=127.6.98.84:37985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.98.126:39627 with env {}
W20240503 13:55:38.534794 6549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:55:38.535044 6549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:55:38.535086 6549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:55:38.538167 6549 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 13:55:38.538230 6549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:55:38.538249 6549 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 13:55:38.538265 6549 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 13:55:38.542094 6549 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.98.84:37985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.98.126:39627
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.98.126:39627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.6.98.126
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.6549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:55:38.543030 6549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:55:38.544171 6549 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:55:38.552086 6554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:55:38.552634 6555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:55:38.552634 6557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:55:38.553148 6549 server_base.cc:1034] running on GCE node
I20240503 13:55:38.553655 6549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:55:38.554565 6549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:55:38.555720 6549 hybrid_clock.cc:648] HybridClock initialized: now 1714744538555696 us; error 39 us; skew 500 ppm
I20240503 13:55:38.562624 6549 webserver.cc:460] Webserver started at http://127.6.98.126:37791/ using document root <none> and password file <none>
I20240503 13:55:38.563225 6549 fs_manager.cc:359] Metadata directory not provided
I20240503 13:55:38.563290 6549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:55:38.563486 6549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:55:38.565346 6549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "de9fcd5581d746f6b0fc8aa5064a2a2a"
format_stamp: "Formatted at 2024-05-03 13:55:38 on dist-test-slave-zg9z"
I20240503 13:55:38.565837 6549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "de9fcd5581d746f6b0fc8aa5064a2a2a"
format_stamp: "Formatted at 2024-05-03 13:55:38 on dist-test-slave-zg9z"
I20240503 13:55:38.569329 6549 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 13:55:38.571470 6564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:55:38.572461 6549 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:55:38.572615 6549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/master-0/wal
uuid: "de9fcd5581d746f6b0fc8aa5064a2a2a"
format_stamp: "Formatted at 2024-05-03 13:55:38 on dist-test-slave-zg9z"
I20240503 13:55:38.572702 6549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:55:38.583189 6549 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:55:38.583842 6549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:55:38.583997 6549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:55:38.590771 6617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.98.126:39627 every 8 connection(s)
I20240503 13:55:38.590768 6549 rpc_server.cc:307] RPC server started. Bound to: 127.6.98.126:39627
I20240503 13:55:38.591842 6549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 13:55:38.592593 6537 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu as pid 6549
I20240503 13:55:38.592712 6537 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 13:55:38.594542 6618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:55:38.599929 6618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de9fcd5581d746f6b0fc8aa5064a2a2a: Bootstrap starting.
I20240503 13:55:38.602005 6618 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de9fcd5581d746f6b0fc8aa5064a2a2a: Neither blocks nor log segments found. Creating new log.
I20240503 13:55:38.602677 6618 log.cc:824] T 00000000000000000000000000000000 P de9fcd5581d746f6b0fc8aa5064a2a2a: Log is configured to *not* fsync() on all Append() calls
I20240503 13:55:38.604467 6618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de9fcd5581d746f6b0fc8aa5064a2a2a: No bootstrap required, opened a new log
I20240503 13:55:38.607152 6618 raft_consensus.cc:348] T 00000000000000000000000000000000 P de9fcd5581d746f6b0fc8aa5064a2a2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de9fcd5581d746f6b0fc8aa5064a2a2a" member_type: VOTER last_known_addr { host: "127.6.98.126" port: 39627 } }
I20240503 13:55:38.607384 6618 raft_consensus.cc:374] T 00000000000000000000000000000000 P de9fcd5581d746f6b0fc8aa5064a2a2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:55:38.607460 6618 raft_consensus.cc:729] T 00000000000000000000000000000000 P de9fcd5581d746f6b0fc8aa5064a2a2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de9fcd5581d746f6b0fc8aa5064a2a2a, State: Initialized, Role: FOLLOWER
I20240503 13:55:38.607860 6618 consensus_queue.cc:260] T 00000000000000000000000000000000 P de9fcd5581d746f6b0fc8aa5064a2a2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de9fcd5581d746f6b0fc8aa5064a2a2a" member_type: VOTER last_known_addr { host: "127.6.98.126" port: 39627 } }
I20240503 13:55:38.607981 6618 raft_consensus.cc:388] T 00000000000000000000000000000000 P de9fcd5581d746f6b0fc8aa5064a2a2a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:55:38.608026 6618 raft_consensus.cc:482] T 00000000000000000000000000000000 P de9fcd5581d746f6b0fc8aa5064a2a2a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:55:38.608101 6618 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de9fcd5581d746f6b0fc8aa5064a2a2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:55:38.608896 6618 raft_consensus.cc:504] T 00000000000000000000000000000000 P de9fcd5581d746f6b0fc8aa5064a2a2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de9fcd5581d746f6b0fc8aa5064a2a2a" member_type: VOTER last_known_addr { host: "127.6.98.126" port: 39627 } }
I20240503 13:55:38.609194 6618 leader_election.cc:302] T 00000000000000000000000000000000 P de9fcd5581d746f6b0fc8aa5064a2a2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de9fcd5581d746f6b0fc8aa5064a2a2a; no voters:
I20240503 13:55:38.609442 6618 leader_election.cc:288] T 00000000000000000000000000000000 P de9fcd5581d746f6b0fc8aa5064a2a2a [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 13:55:38.609767 6623 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de9fcd5581d746f6b0fc8aa5064a2a2a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:55:38.609966 6623 raft_consensus.cc:686] T 00000000000000000000000000000000 P de9fcd5581d746f6b0fc8aa5064a2a2a [term 1 LEADER]: Becoming Leader. State: Replica: de9fcd5581d746f6b0fc8aa5064a2a2a, State: Running, Role: LEADER
I20240503 13:55:38.610294 6623 consensus_queue.cc:237] T 00000000000000000000000000000000 P de9fcd5581d746f6b0fc8aa5064a2a2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de9fcd5581d746f6b0fc8aa5064a2a2a" member_type: VOTER last_known_addr { host: "127.6.98.126" port: 39627 } }
I20240503 13:55:38.610572 6618 sys_catalog.cc:564] T 00000000000000000000000000000000 P de9fcd5581d746f6b0fc8aa5064a2a2a [sys.catalog]: configured and running, proceeding with master startup.
I20240503 13:55:38.611759 6624 sys_catalog.cc:455] T 00000000000000000000000000000000 P de9fcd5581d746f6b0fc8aa5064a2a2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de9fcd5581d746f6b0fc8aa5064a2a2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de9fcd5581d746f6b0fc8aa5064a2a2a" member_type: VOTER last_known_addr { host: "127.6.98.126" port: 39627 } } }
I20240503 13:55:38.611801 6625 sys_catalog.cc:455] T 00000000000000000000000000000000 P de9fcd5581d746f6b0fc8aa5064a2a2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader de9fcd5581d746f6b0fc8aa5064a2a2a. Latest consensus state: current_term: 1 leader_uuid: "de9fcd5581d746f6b0fc8aa5064a2a2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de9fcd5581d746f6b0fc8aa5064a2a2a" member_type: VOTER last_known_addr { host: "127.6.98.126" port: 39627 } } }
I20240503 13:55:38.611869 6624 sys_catalog.cc:458] T 00000000000000000000000000000000 P de9fcd5581d746f6b0fc8aa5064a2a2a [sys.catalog]: This master's current role is: LEADER
I20240503 13:55:38.611903 6625 sys_catalog.cc:458] T 00000000000000000000000000000000 P de9fcd5581d746f6b0fc8aa5064a2a2a [sys.catalog]: This master's current role is: LEADER
I20240503 13:55:38.612228 6632 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 13:55:38.614918 6632 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 13:55:38.619887 6632 catalog_manager.cc:1347] Generated new cluster ID: 64f9eb25105d4a88b0bdd62ef2fd77e8
I20240503 13:55:38.619961 6632 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 13:55:38.636320 6632 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 13:55:38.637270 6632 catalog_manager.cc:1504] Loading token signing keys...
I20240503 13:55:38.643209 6632 catalog_manager.cc:5857] T 00000000000000000000000000000000 P de9fcd5581d746f6b0fc8aa5064a2a2a: Generated new TSK 0
I20240503 13:55:38.644032 6632 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 13:55:38.658598 6537 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
/tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.98.65:0
--local_ip_for_outbound_sockets=127.6.98.65
--webserver_interface=127.6.98.65
--webserver_port=0
--tserver_master_addrs=127.6.98.126:39627
--builtin_ntp_servers=127.6.98.84:37985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20240503 13:55:38.767407 6642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:55:38.767637 6642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:55:38.767679 6642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:55:38.770761 6642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:55:38.770874 6642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.98.65
I20240503 13:55:38.774605 6642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.98.84:37985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.98.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.6.98.65
--webserver_port=0
--tserver_master_addrs=127.6.98.126:39627
--never_fsync=true
--heap_profile_path=/tmp/kudu.6642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.98.65
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:55:38.775506 6642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:55:38.776672 6642 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:55:38.785009 6650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:55:38.785259 6647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:55:38.785451 6648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:55:38.785953 6642 server_base.cc:1034] running on GCE node
I20240503 13:55:38.786326 6642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:55:38.786906 6642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:55:38.788067 6642 hybrid_clock.cc:648] HybridClock initialized: now 1714744538788039 us; error 41 us; skew 500 ppm
I20240503 13:55:38.789916 6642 webserver.cc:460] Webserver started at http://127.6.98.65:34761/ using document root <none> and password file <none>
I20240503 13:55:38.790449 6642 fs_manager.cc:359] Metadata directory not provided
I20240503 13:55:38.790508 6642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:55:38.790675 6642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:55:38.792354 6642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "0e939678cbfc4239b6bca7e32c488cb9"
format_stamp: "Formatted at 2024-05-03 13:55:38 on dist-test-slave-zg9z"
I20240503 13:55:38.792845 6642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "0e939678cbfc4239b6bca7e32c488cb9"
format_stamp: "Formatted at 2024-05-03 13:55:38 on dist-test-slave-zg9z"
I20240503 13:55:38.796162 6642 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 13:55:38.798384 6656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:55:38.799350 6642 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240503 13:55:38.799453 6642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "0e939678cbfc4239b6bca7e32c488cb9"
format_stamp: "Formatted at 2024-05-03 13:55:38 on dist-test-slave-zg9z"
I20240503 13:55:38.799535 6642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:55:38.829674 6642 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:55:38.830394 6642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:55:38.830562 6642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:55:38.831130 6642 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:55:38.832046 6642 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:55:38.832093 6642 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:55:38.832144 6642 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:55:38.832172 6642 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:55:38.840914 6642 rpc_server.cc:307] RPC server started. Bound to: 127.6.98.65:44017
I20240503 13:55:38.841032 6769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.98.65:44017 every 8 connection(s)
I20240503 13:55:38.841919 6642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 13:55:38.844666 6537 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu as pid 6642
I20240503 13:55:38.844786 6537 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 13:55:38.847110 6537 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
/tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.98.66:0
--local_ip_for_outbound_sockets=127.6.98.66
--webserver_interface=127.6.98.66
--webserver_port=0
--tserver_master_addrs=127.6.98.126:39627
--builtin_ntp_servers=127.6.98.84:37985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240503 13:55:38.851922 6770 heartbeater.cc:347] Connected to a master server at 127.6.98.126:39627
I20240503 13:55:38.852212 6770 heartbeater.cc:464] Registering TS with master...
I20240503 13:55:38.853005 6770 heartbeater.cc:511] Master 127.6.98.126:39627 requested a full tablet report, sending...
I20240503 13:55:38.854693 6581 ts_manager.cc:194] Registered new tserver with Master: 0e939678cbfc4239b6bca7e32c488cb9 (127.6.98.65:44017)
I20240503 13:55:38.855890 6581 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.98.65:57043
W20240503 13:55:38.955833 6773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:55:38.956064 6773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:55:38.956108 6773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:55:38.959213 6773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:55:38.959331 6773 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.98.66
I20240503 13:55:38.962998 6773 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.98.84:37985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.98.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.6.98.66
--webserver_port=0
--tserver_master_addrs=127.6.98.126:39627
--never_fsync=true
--heap_profile_path=/tmp/kudu.6773
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.98.66
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:55:38.963912 6773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:55:38.965250 6773 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:55:38.973315 6779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:55:38.973332 6781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:55:38.973598 6778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:55:38.974149 6773 server_base.cc:1034] running on GCE node
I20240503 13:55:38.974509 6773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:55:38.975049 6773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:55:38.976187 6773 hybrid_clock.cc:648] HybridClock initialized: now 1714744538976164 us; error 37 us; skew 500 ppm
I20240503 13:55:38.977994 6773 webserver.cc:460] Webserver started at http://127.6.98.66:39861/ using document root <none> and password file <none>
I20240503 13:55:38.978504 6773 fs_manager.cc:359] Metadata directory not provided
I20240503 13:55:38.978550 6773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:55:38.978696 6773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:55:38.980298 6773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "f0f3ad90d22447e5ae6f815d0813d0d6"
format_stamp: "Formatted at 2024-05-03 13:55:38 on dist-test-slave-zg9z"
I20240503 13:55:38.980751 6773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "f0f3ad90d22447e5ae6f815d0813d0d6"
format_stamp: "Formatted at 2024-05-03 13:55:38 on dist-test-slave-zg9z"
I20240503 13:55:38.983872 6773 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20240503 13:55:38.986104 6788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:55:38.987102 6773 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 13:55:38.987208 6773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "f0f3ad90d22447e5ae6f815d0813d0d6"
format_stamp: "Formatted at 2024-05-03 13:55:38 on dist-test-slave-zg9z"
I20240503 13:55:38.987288 6773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:55:39.000847 6773 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:55:39.001494 6773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:55:39.001633 6773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:55:39.002140 6773 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:55:39.003096 6773 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:55:39.003144 6773 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:55:39.003183 6773 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:55:39.003202 6773 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:55:39.011617 6773 rpc_server.cc:307] RPC server started. Bound to: 127.6.98.66:46431
I20240503 13:55:39.011646 6903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.98.66:46431 every 8 connection(s)
I20240503 13:55:39.012643 6773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 13:55:39.012916 6537 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu as pid 6773
I20240503 13:55:39.013067 6537 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 13:55:39.015391 6537 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
/tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.98.67:0
--local_ip_for_outbound_sockets=127.6.98.67
--webserver_interface=127.6.98.67
--webserver_port=0
--tserver_master_addrs=127.6.98.126:39627
--builtin_ntp_servers=127.6.98.84:37985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240503 13:55:39.022842 6904 heartbeater.cc:347] Connected to a master server at 127.6.98.126:39627
I20240503 13:55:39.023118 6904 heartbeater.cc:464] Registering TS with master...
I20240503 13:55:39.023808 6904 heartbeater.cc:511] Master 127.6.98.126:39627 requested a full tablet report, sending...
I20240503 13:55:39.024924 6581 ts_manager.cc:194] Registered new tserver with Master: f0f3ad90d22447e5ae6f815d0813d0d6 (127.6.98.66:46431)
I20240503 13:55:39.025488 6581 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.98.66:59711
W20240503 13:55:39.124083 6907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:55:39.124305 6907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:55:39.124364 6907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:55:39.127427 6907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:55:39.127554 6907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.98.67
I20240503 13:55:39.131147 6907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.98.84:37985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.98.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.6.98.67
--webserver_port=0
--tserver_master_addrs=127.6.98.126:39627
--never_fsync=true
--heap_profile_path=/tmp/kudu.6907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.98.67
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:55:39.132035 6907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:55:39.133198 6907 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:55:39.140868 6912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:55:39.141321 6907 server_base.cc:1034] running on GCE node
W20240503 13:55:39.141588 6913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:55:39.141903 6915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:55:39.142277 6907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:55:39.142814 6907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:55:39.143955 6907 hybrid_clock.cc:648] HybridClock initialized: now 1714744539143931 us; error 38 us; skew 500 ppm
I20240503 13:55:39.145938 6907 webserver.cc:460] Webserver started at http://127.6.98.67:37117/ using document root <none> and password file <none>
I20240503 13:55:39.146479 6907 fs_manager.cc:359] Metadata directory not provided
I20240503 13:55:39.146529 6907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:55:39.146677 6907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:55:39.148304 6907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "748b9a0a333b4bb98ed3e03850ad0339"
format_stamp: "Formatted at 2024-05-03 13:55:39 on dist-test-slave-zg9z"
I20240503 13:55:39.148792 6907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "748b9a0a333b4bb98ed3e03850ad0339"
format_stamp: "Formatted at 2024-05-03 13:55:39 on dist-test-slave-zg9z"
I20240503 13:55:39.152308 6907 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 13:55:39.154608 6921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:55:39.155628 6907 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:55:39.155725 6907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "748b9a0a333b4bb98ed3e03850ad0339"
format_stamp: "Formatted at 2024-05-03 13:55:39 on dist-test-slave-zg9z"
I20240503 13:55:39.155804 6907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:55:39.168227 6907 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:55:39.168885 6907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:55:39.169025 6907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:55:39.169510 6907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:55:39.170362 6907 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:55:39.170405 6907 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:55:39.170442 6907 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:55:39.170464 6907 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:55:39.178857 6907 rpc_server.cc:307] RPC server started. Bound to: 127.6.98.67:44215
I20240503 13:55:39.178929 7035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.98.67:44215 every 8 connection(s)
I20240503 13:55:39.179778 6907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 13:55:39.181797 6537 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu as pid 6907
I20240503 13:55:39.181880 6537 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 13:55:39.184053 6537 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
/tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.98.68:0
--local_ip_for_outbound_sockets=127.6.98.68
--webserver_interface=127.6.98.68
--webserver_port=0
--tserver_master_addrs=127.6.98.126:39627
--builtin_ntp_servers=127.6.98.84:37985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240503 13:55:39.188916 7036 heartbeater.cc:347] Connected to a master server at 127.6.98.126:39627
I20240503 13:55:39.189184 7036 heartbeater.cc:464] Registering TS with master...
I20240503 13:55:39.189880 7036 heartbeater.cc:511] Master 127.6.98.126:39627 requested a full tablet report, sending...
I20240503 13:55:39.191012 6581 ts_manager.cc:194] Registered new tserver with Master: 748b9a0a333b4bb98ed3e03850ad0339 (127.6.98.67:44215)
I20240503 13:55:39.191606 6581 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.98.67:45585
W20240503 13:55:39.292273 7039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:55:39.292554 7039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:55:39.292630 7039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:55:39.295804 7039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:55:39.295943 7039 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.98.68
I20240503 13:55:39.299748 7039 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.98.84:37985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.98.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.6.98.68
--webserver_port=0
--tserver_master_addrs=127.6.98.126:39627
--never_fsync=true
--heap_profile_path=/tmp/kudu.7039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.98.68
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:55:39.300733 7039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:55:39.301880 7039 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:55:39.309924 7047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:55:39.310238 7044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:55:39.310554 7045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:55:39.311142 7039 server_base.cc:1034] running on GCE node
I20240503 13:55:39.311511 7039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:55:39.312103 7039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:55:39.313263 7039 hybrid_clock.cc:648] HybridClock initialized: now 1714744539313249 us; error 28 us; skew 500 ppm
I20240503 13:55:39.315106 7039 webserver.cc:460] Webserver started at http://127.6.98.68:38699/ using document root <none> and password file <none>
I20240503 13:55:39.315663 7039 fs_manager.cc:359] Metadata directory not provided
I20240503 13:55:39.315721 7039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:55:39.315876 7039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:55:39.317620 7039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "c63fb818a2894e77803f588fb3577c2c"
format_stamp: "Formatted at 2024-05-03 13:55:39 on dist-test-slave-zg9z"
I20240503 13:55:39.318081 7039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "c63fb818a2894e77803f588fb3577c2c"
format_stamp: "Formatted at 2024-05-03 13:55:39 on dist-test-slave-zg9z"
I20240503 13:55:39.321478 7039 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 13:55:39.323668 7054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:55:39.324734 7039 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 13:55:39.324829 7039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "c63fb818a2894e77803f588fb3577c2c"
format_stamp: "Formatted at 2024-05-03 13:55:39 on dist-test-slave-zg9z"
I20240503 13:55:39.324910 7039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:55:39.339919 7039 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:55:39.340632 7039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:55:39.340765 7039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:55:39.341262 7039 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:55:39.342121 7039 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:55:39.342166 7039 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:55:39.342206 7039 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:55:39.342226 7039 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:55:39.351183 7039 rpc_server.cc:307] RPC server started. Bound to: 127.6.98.68:35259
I20240503 13:55:39.351215 7167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.98.68:35259 every 8 connection(s)
I20240503 13:55:39.352108 7039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 13:55:39.360143 6537 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu as pid 7039
I20240503 13:55:39.360257 6537 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 13:55:39.360867 7168 heartbeater.cc:347] Connected to a master server at 127.6.98.126:39627
I20240503 13:55:39.361112 7168 heartbeater.cc:464] Registering TS with master...
I20240503 13:55:39.361723 7168 heartbeater.cc:511] Master 127.6.98.126:39627 requested a full tablet report, sending...
I20240503 13:55:39.362440 6537 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
/tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.98.69:0
--local_ip_for_outbound_sockets=127.6.98.69
--webserver_interface=127.6.98.69
--webserver_port=0
--tserver_master_addrs=127.6.98.126:39627
--builtin_ntp_servers=127.6.98.84:37985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240503 13:55:39.362630 6581 ts_manager.cc:194] Registered new tserver with Master: c63fb818a2894e77803f588fb3577c2c (127.6.98.68:35259)
I20240503 13:55:39.363155 6581 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.98.68:41841
W20240503 13:55:39.467700 7171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:55:39.467931 7171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:55:39.467998 7171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:55:39.471060 7171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:55:39.471187 7171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.98.69
I20240503 13:55:39.474782 7171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.98.84:37985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.98.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.6.98.69
--webserver_port=0
--tserver_master_addrs=127.6.98.126:39627
--never_fsync=true
--heap_profile_path=/tmp/kudu.7171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.98.69
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:55:39.475677 7171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:55:39.476827 7171 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:55:39.484892 7177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:55:39.485378 7179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:55:39.485478 7176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:55:39.485837 7171 server_base.cc:1034] running on GCE node
I20240503 13:55:39.486207 7171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:55:39.486742 7171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:55:39.487922 7171 hybrid_clock.cc:648] HybridClock initialized: now 1714744539487905 us; error 28 us; skew 500 ppm
I20240503 13:55:39.489800 7171 webserver.cc:460] Webserver started at http://127.6.98.69:44017/ using document root <none> and password file <none>
I20240503 13:55:39.490427 7171 fs_manager.cc:359] Metadata directory not provided
I20240503 13:55:39.490480 7171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:55:39.490645 7171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:55:39.492312 7171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "0c6667ca7a0842909329aa1929527286"
format_stamp: "Formatted at 2024-05-03 13:55:39 on dist-test-slave-zg9z"
I20240503 13:55:39.492789 7171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "0c6667ca7a0842909329aa1929527286"
format_stamp: "Formatted at 2024-05-03 13:55:39 on dist-test-slave-zg9z"
I20240503 13:55:39.495999 7171 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 13:55:39.498242 7185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:55:39.499330 7171 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240503 13:55:39.499437 7171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "0c6667ca7a0842909329aa1929527286"
format_stamp: "Formatted at 2024-05-03 13:55:39 on dist-test-slave-zg9z"
I20240503 13:55:39.499511 7171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:55:39.514848 7171 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:55:39.515558 7171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:55:39.515693 7171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:55:39.516219 7171 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:55:39.517130 7171 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:55:39.517175 7171 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:55:39.517213 7171 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:55:39.517233 7171 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:55:39.525249 7171 rpc_server.cc:307] RPC server started. Bound to: 127.6.98.69:41057
I20240503 13:55:39.525279 7300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.98.69:41057 every 8 connection(s)
I20240503 13:55:39.526145 7171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 13:55:39.527870 6537 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu as pid 7171
I20240503 13:55:39.527976 6537 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 13:55:39.534677 7301 heartbeater.cc:347] Connected to a master server at 127.6.98.126:39627
I20240503 13:55:39.534895 7301 heartbeater.cc:464] Registering TS with master...
I20240503 13:55:39.535423 7301 heartbeater.cc:511] Master 127.6.98.126:39627 requested a full tablet report, sending...
I20240503 13:55:39.536338 6581 ts_manager.cc:194] Registered new tserver with Master: 0c6667ca7a0842909329aa1929527286 (127.6.98.69:41057)
I20240503 13:55:39.536886 6581 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.98.69:41555
I20240503 13:55:39.542359 6537 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 13:55:39.554138 6581 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50158:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 13:55:39.572841 6970 tablet_service.cc:1467] Processing CreateTablet for tablet 8ec70fa5019345dc8aa9657aeb49189a (DEFAULT_TABLE table=TestTable [id=0095c9a0f2524cf3829b647d70e3711f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 13:55:39.572837 6837 tablet_service.cc:1467] Processing CreateTablet for tablet 8ec70fa5019345dc8aa9657aeb49189a (DEFAULT_TABLE table=TestTable [id=0095c9a0f2524cf3829b647d70e3711f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 13:55:39.573055 7233 tablet_service.cc:1467] Processing CreateTablet for tablet 8ec70fa5019345dc8aa9657aeb49189a (DEFAULT_TABLE table=TestTable [id=0095c9a0f2524cf3829b647d70e3711f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 13:55:39.573851 7233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ec70fa5019345dc8aa9657aeb49189a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:55:39.573855 6837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ec70fa5019345dc8aa9657aeb49189a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:55:39.573855 6970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ec70fa5019345dc8aa9657aeb49189a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:55:39.579118 7318 tablet_bootstrap.cc:492] T 8ec70fa5019345dc8aa9657aeb49189a P 0c6667ca7a0842909329aa1929527286: Bootstrap starting.
I20240503 13:55:39.579910 7320 tablet_bootstrap.cc:492] T 8ec70fa5019345dc8aa9657aeb49189a P 748b9a0a333b4bb98ed3e03850ad0339: Bootstrap starting.
I20240503 13:55:39.580886 7319 tablet_bootstrap.cc:492] T 8ec70fa5019345dc8aa9657aeb49189a P f0f3ad90d22447e5ae6f815d0813d0d6: Bootstrap starting.
I20240503 13:55:39.581444 7318 tablet_bootstrap.cc:654] T 8ec70fa5019345dc8aa9657aeb49189a P 0c6667ca7a0842909329aa1929527286: Neither blocks nor log segments found. Creating new log.
I20240503 13:55:39.581827 7320 tablet_bootstrap.cc:654] T 8ec70fa5019345dc8aa9657aeb49189a P 748b9a0a333b4bb98ed3e03850ad0339: Neither blocks nor log segments found. Creating new log.
I20240503 13:55:39.582190 7318 log.cc:824] T 8ec70fa5019345dc8aa9657aeb49189a P 0c6667ca7a0842909329aa1929527286: Log is configured to *not* fsync() on all Append() calls
I20240503 13:55:39.582592 7320 log.cc:824] T 8ec70fa5019345dc8aa9657aeb49189a P 748b9a0a333b4bb98ed3e03850ad0339: Log is configured to *not* fsync() on all Append() calls
I20240503 13:55:39.583243 7319 tablet_bootstrap.cc:654] T 8ec70fa5019345dc8aa9657aeb49189a P f0f3ad90d22447e5ae6f815d0813d0d6: Neither blocks nor log segments found. Creating new log.
I20240503 13:55:39.583767 7318 tablet_bootstrap.cc:492] T 8ec70fa5019345dc8aa9657aeb49189a P 0c6667ca7a0842909329aa1929527286: No bootstrap required, opened a new log
I20240503 13:55:39.583937 7318 ts_tablet_manager.cc:1367] T 8ec70fa5019345dc8aa9657aeb49189a P 0c6667ca7a0842909329aa1929527286: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 13:55:39.583957 7319 log.cc:824] T 8ec70fa5019345dc8aa9657aeb49189a P f0f3ad90d22447e5ae6f815d0813d0d6: Log is configured to *not* fsync() on all Append() calls
I20240503 13:55:39.584252 7320 tablet_bootstrap.cc:492] T 8ec70fa5019345dc8aa9657aeb49189a P 748b9a0a333b4bb98ed3e03850ad0339: No bootstrap required, opened a new log
I20240503 13:55:39.584563 7320 ts_tablet_manager.cc:1367] T 8ec70fa5019345dc8aa9657aeb49189a P 748b9a0a333b4bb98ed3e03850ad0339: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20240503 13:55:39.585825 7319 tablet_bootstrap.cc:492] T 8ec70fa5019345dc8aa9657aeb49189a P f0f3ad90d22447e5ae6f815d0813d0d6: No bootstrap required, opened a new log
I20240503 13:55:39.586002 7319 ts_tablet_manager.cc:1367] T 8ec70fa5019345dc8aa9657aeb49189a P f0f3ad90d22447e5ae6f815d0813d0d6: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 13:55:39.587441 7320 raft_consensus.cc:348] T 8ec70fa5019345dc8aa9657aeb49189a P 748b9a0a333b4bb98ed3e03850ad0339 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0f3ad90d22447e5ae6f815d0813d0d6" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 46431 } } peers { permanent_uuid: "0c6667ca7a0842909329aa1929527286" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 41057 } } peers { permanent_uuid: "748b9a0a333b4bb98ed3e03850ad0339" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 44215 } }
I20240503 13:55:39.587600 7320 raft_consensus.cc:374] T 8ec70fa5019345dc8aa9657aeb49189a P 748b9a0a333b4bb98ed3e03850ad0339 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:55:39.587473 7318 raft_consensus.cc:348] T 8ec70fa5019345dc8aa9657aeb49189a P 0c6667ca7a0842909329aa1929527286 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0f3ad90d22447e5ae6f815d0813d0d6" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 46431 } } peers { permanent_uuid: "0c6667ca7a0842909329aa1929527286" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 41057 } } peers { permanent_uuid: "748b9a0a333b4bb98ed3e03850ad0339" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 44215 } }
I20240503 13:55:39.587631 7320 raft_consensus.cc:729] T 8ec70fa5019345dc8aa9657aeb49189a P 748b9a0a333b4bb98ed3e03850ad0339 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 748b9a0a333b4bb98ed3e03850ad0339, State: Initialized, Role: FOLLOWER
I20240503 13:55:39.587658 7318 raft_consensus.cc:374] T 8ec70fa5019345dc8aa9657aeb49189a P 0c6667ca7a0842909329aa1929527286 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:55:39.587702 7318 raft_consensus.cc:729] T 8ec70fa5019345dc8aa9657aeb49189a P 0c6667ca7a0842909329aa1929527286 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c6667ca7a0842909329aa1929527286, State: Initialized, Role: FOLLOWER
I20240503 13:55:39.588011 7320 consensus_queue.cc:260] T 8ec70fa5019345dc8aa9657aeb49189a P 748b9a0a333b4bb98ed3e03850ad0339 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0f3ad90d22447e5ae6f815d0813d0d6" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 46431 } } peers { permanent_uuid: "0c6667ca7a0842909329aa1929527286" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 41057 } } peers { permanent_uuid: "748b9a0a333b4bb98ed3e03850ad0339" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 44215 } }
I20240503 13:55:39.588109 7318 consensus_queue.cc:260] T 8ec70fa5019345dc8aa9657aeb49189a P 0c6667ca7a0842909329aa1929527286 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0f3ad90d22447e5ae6f815d0813d0d6" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 46431 } } peers { permanent_uuid: "0c6667ca7a0842909329aa1929527286" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 41057 } } peers { permanent_uuid: "748b9a0a333b4bb98ed3e03850ad0339" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 44215 } }
I20240503 13:55:39.588891 7320 ts_tablet_manager.cc:1398] T 8ec70fa5019345dc8aa9657aeb49189a P 748b9a0a333b4bb98ed3e03850ad0339: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240503 13:55:39.588907 7318 ts_tablet_manager.cc:1398] T 8ec70fa5019345dc8aa9657aeb49189a P 0c6667ca7a0842909329aa1929527286: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240503 13:55:39.589012 7319 raft_consensus.cc:348] T 8ec70fa5019345dc8aa9657aeb49189a P f0f3ad90d22447e5ae6f815d0813d0d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0f3ad90d22447e5ae6f815d0813d0d6" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 46431 } } peers { permanent_uuid: "0c6667ca7a0842909329aa1929527286" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 41057 } } peers { permanent_uuid: "748b9a0a333b4bb98ed3e03850ad0339" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 44215 } }
I20240503 13:55:39.589238 7036 heartbeater.cc:503] Master 127.6.98.126:39627 was elected leader, sending a full tablet report...
I20240503 13:55:39.589249 7301 heartbeater.cc:503] Master 127.6.98.126:39627 was elected leader, sending a full tablet report...
I20240503 13:55:39.589247 7319 raft_consensus.cc:374] T 8ec70fa5019345dc8aa9657aeb49189a P f0f3ad90d22447e5ae6f815d0813d0d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:55:39.589298 7319 raft_consensus.cc:729] T 8ec70fa5019345dc8aa9657aeb49189a P f0f3ad90d22447e5ae6f815d0813d0d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0f3ad90d22447e5ae6f815d0813d0d6, State: Initialized, Role: FOLLOWER
I20240503 13:55:39.589709 7319 consensus_queue.cc:260] T 8ec70fa5019345dc8aa9657aeb49189a P f0f3ad90d22447e5ae6f815d0813d0d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0f3ad90d22447e5ae6f815d0813d0d6" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 46431 } } peers { permanent_uuid: "0c6667ca7a0842909329aa1929527286" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 41057 } } peers { permanent_uuid: "748b9a0a333b4bb98ed3e03850ad0339" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 44215 } }
I20240503 13:55:39.590499 7319 ts_tablet_manager.cc:1398] T 8ec70fa5019345dc8aa9657aeb49189a P f0f3ad90d22447e5ae6f815d0813d0d6: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240503 13:55:39.590862 6904 heartbeater.cc:503] Master 127.6.98.126:39627 was elected leader, sending a full tablet report...
I20240503 13:55:39.599521 7326 raft_consensus.cc:482] T 8ec70fa5019345dc8aa9657aeb49189a P f0f3ad90d22447e5ae6f815d0813d0d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:55:39.599696 7326 raft_consensus.cc:504] T 8ec70fa5019345dc8aa9657aeb49189a P f0f3ad90d22447e5ae6f815d0813d0d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0f3ad90d22447e5ae6f815d0813d0d6" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 46431 } } peers { permanent_uuid: "0c6667ca7a0842909329aa1929527286" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 41057 } } peers { permanent_uuid: "748b9a0a333b4bb98ed3e03850ad0339" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 44215 } }
I20240503 13:55:39.600773 7326 leader_election.cc:288] T 8ec70fa5019345dc8aa9657aeb49189a P f0f3ad90d22447e5ae6f815d0813d0d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c6667ca7a0842909329aa1929527286 (127.6.98.69:41057), 748b9a0a333b4bb98ed3e03850ad0339 (127.6.98.67:44215)
I20240503 13:55:39.605108 6990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec70fa5019345dc8aa9657aeb49189a" candidate_uuid: "f0f3ad90d22447e5ae6f815d0813d0d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "748b9a0a333b4bb98ed3e03850ad0339" is_pre_election: true
I20240503 13:55:39.605381 7255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec70fa5019345dc8aa9657aeb49189a" candidate_uuid: "f0f3ad90d22447e5ae6f815d0813d0d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c6667ca7a0842909329aa1929527286" is_pre_election: true
I20240503 13:55:39.605418 6990 raft_consensus.cc:2454] T 8ec70fa5019345dc8aa9657aeb49189a P 748b9a0a333b4bb98ed3e03850ad0339 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0f3ad90d22447e5ae6f815d0813d0d6 in term 0.
I20240503 13:55:39.605630 7255 raft_consensus.cc:2454] T 8ec70fa5019345dc8aa9657aeb49189a P 0c6667ca7a0842909329aa1929527286 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0f3ad90d22447e5ae6f815d0813d0d6 in term 0.
I20240503 13:55:39.605873 6789 leader_election.cc:302] T 8ec70fa5019345dc8aa9657aeb49189a P f0f3ad90d22447e5ae6f815d0813d0d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 748b9a0a333b4bb98ed3e03850ad0339, f0f3ad90d22447e5ae6f815d0813d0d6; no voters:
I20240503 13:55:39.606123 7326 raft_consensus.cc:2781] T 8ec70fa5019345dc8aa9657aeb49189a P f0f3ad90d22447e5ae6f815d0813d0d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:55:39.606194 7326 raft_consensus.cc:482] T 8ec70fa5019345dc8aa9657aeb49189a P f0f3ad90d22447e5ae6f815d0813d0d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:55:39.606238 7326 raft_consensus.cc:3037] T 8ec70fa5019345dc8aa9657aeb49189a P f0f3ad90d22447e5ae6f815d0813d0d6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:55:39.607064 7326 raft_consensus.cc:504] T 8ec70fa5019345dc8aa9657aeb49189a P f0f3ad90d22447e5ae6f815d0813d0d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0f3ad90d22447e5ae6f815d0813d0d6" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 46431 } } peers { permanent_uuid: "0c6667ca7a0842909329aa1929527286" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 41057 } } peers { permanent_uuid: "748b9a0a333b4bb98ed3e03850ad0339" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 44215 } }
I20240503 13:55:39.607370 7326 leader_election.cc:288] T 8ec70fa5019345dc8aa9657aeb49189a P f0f3ad90d22447e5ae6f815d0813d0d6 [CANDIDATE]: Term 1 election: Requested vote from peers 0c6667ca7a0842909329aa1929527286 (127.6.98.69:41057), 748b9a0a333b4bb98ed3e03850ad0339 (127.6.98.67:44215)
I20240503 13:55:39.607683 7255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec70fa5019345dc8aa9657aeb49189a" candidate_uuid: "f0f3ad90d22447e5ae6f815d0813d0d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c6667ca7a0842909329aa1929527286"
I20240503 13:55:39.607740 6990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec70fa5019345dc8aa9657aeb49189a" candidate_uuid: "f0f3ad90d22447e5ae6f815d0813d0d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "748b9a0a333b4bb98ed3e03850ad0339"
I20240503 13:55:39.607789 7255 raft_consensus.cc:3037] T 8ec70fa5019345dc8aa9657aeb49189a P 0c6667ca7a0842909329aa1929527286 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:55:39.607851 6990 raft_consensus.cc:3037] T 8ec70fa5019345dc8aa9657aeb49189a P 748b9a0a333b4bb98ed3e03850ad0339 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:55:39.608665 7255 raft_consensus.cc:2454] T 8ec70fa5019345dc8aa9657aeb49189a P 0c6667ca7a0842909329aa1929527286 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0f3ad90d22447e5ae6f815d0813d0d6 in term 1.
I20240503 13:55:39.608716 6990 raft_consensus.cc:2454] T 8ec70fa5019345dc8aa9657aeb49189a P 748b9a0a333b4bb98ed3e03850ad0339 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0f3ad90d22447e5ae6f815d0813d0d6 in term 1.
I20240503 13:55:39.609014 6792 leader_election.cc:302] T 8ec70fa5019345dc8aa9657aeb49189a P f0f3ad90d22447e5ae6f815d0813d0d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c6667ca7a0842909329aa1929527286, f0f3ad90d22447e5ae6f815d0813d0d6; no voters:
I20240503 13:55:39.609185 7326 raft_consensus.cc:2781] T 8ec70fa5019345dc8aa9657aeb49189a P f0f3ad90d22447e5ae6f815d0813d0d6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:55:39.609392 7326 raft_consensus.cc:686] T 8ec70fa5019345dc8aa9657aeb49189a P f0f3ad90d22447e5ae6f815d0813d0d6 [term 1 LEADER]: Becoming Leader. State: Replica: f0f3ad90d22447e5ae6f815d0813d0d6, State: Running, Role: LEADER
I20240503 13:55:39.609680 7326 consensus_queue.cc:237] T 8ec70fa5019345dc8aa9657aeb49189a P f0f3ad90d22447e5ae6f815d0813d0d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0f3ad90d22447e5ae6f815d0813d0d6" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 46431 } } peers { permanent_uuid: "0c6667ca7a0842909329aa1929527286" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 41057 } } peers { permanent_uuid: "748b9a0a333b4bb98ed3e03850ad0339" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 44215 } }
I20240503 13:55:39.612864 6580 catalog_manager.cc:5484] T 8ec70fa5019345dc8aa9657aeb49189a P f0f3ad90d22447e5ae6f815d0813d0d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to f0f3ad90d22447e5ae6f815d0813d0d6 (127.6.98.66). New cstate: current_term: 1 leader_uuid: "f0f3ad90d22447e5ae6f815d0813d0d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0f3ad90d22447e5ae6f815d0813d0d6" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 46431 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c6667ca7a0842909329aa1929527286" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 41057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "748b9a0a333b4bb98ed3e03850ad0339" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 44215 } health_report { overall_health: UNKNOWN } } }
I20240503 13:55:39.623842 6537 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 13:55:39.625769 6537 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 0e939678cbfc4239b6bca7e32c488cb9 to finish bootstrapping
I20240503 13:55:39.631544 6537 ts_itest-base.cc:233] Waiting for 1 tablets on tserver f0f3ad90d22447e5ae6f815d0813d0d6 to finish bootstrapping
I20240503 13:55:39.636114 6537 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 748b9a0a333b4bb98ed3e03850ad0339 to finish bootstrapping
I20240503 13:55:39.640870 6537 ts_itest-base.cc:233] Waiting for 0 tablets on tserver c63fb818a2894e77803f588fb3577c2c to finish bootstrapping
I20240503 13:55:39.646270 6537 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 0c6667ca7a0842909329aa1929527286 to finish bootstrapping
W20240503 13:55:39.680835 7037 tablet.cc:2369] T 8ec70fa5019345dc8aa9657aeb49189a P 748b9a0a333b4bb98ed3e03850ad0339: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 13:55:39.764000 6905 tablet.cc:2369] T 8ec70fa5019345dc8aa9657aeb49189a P f0f3ad90d22447e5ae6f815d0813d0d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 13:55:39.776949 7302 tablet.cc:2369] T 8ec70fa5019345dc8aa9657aeb49189a P 0c6667ca7a0842909329aa1929527286: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 13:55:39.832288 7233 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:55:39.833395 6837 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:55:39.834087 6704 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:55:39.834170 7102 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:55:39.835057 6970 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 13:55:39.852859 6765 messenger.cc:188] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.6.98.65:44017
I20240503 13:55:39.858660 6770 heartbeater.cc:503] Master 127.6.98.126:39627 was elected leader, sending a full tablet report...
I20240503 13:55:39.934819 6704 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:55:39.936949 7102 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:55:39.937304 6837 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:55:39.939137 7233 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:55:39.941140 6970 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:55:39.974717 6537 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu with pid 6642
I20240503 13:55:39.981894 6537 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu with pid 6773
I20240503 13:55:39.987354 6537 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu with pid 6907
I20240503 13:55:39.992183 6537 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu with pid 7039
I20240503 13:55:39.997057 6537 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu with pid 7171
I20240503 13:55:40.001668 6537 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu with pid 6549
2024-05-03T13:55:40Z chronyd exiting
[ OK ] AdminCliTest.RebalancerReportOnly (1613 ms)
[----------] 1 test from AdminCliTest (1613 ms total)
[----------] 1 test from RebalanceStartCriteriaTest
[ RUN ] RebalanceStartCriteriaTest.TabletServerIsDown/0
I20240503 13:55:40.023392 6537 test_util.cc:268] Using random seed: 321931867
I20240503 13:55:40.024128 6537 ts_itest-base.cc:112] Starting cluster with:
I20240503 13:55:40.024169 6537 ts_itest-base.cc:113] --------------
I20240503 13:55:40.024183 6537 ts_itest-base.cc:114] 5 tablet servers
I20240503 13:55:40.024197 6537 ts_itest-base.cc:115] 3 replicas per TS
I20240503 13:55:40.024209 6537 ts_itest-base.cc:116] --------------
2024-05-03T13:55:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T13:55:40Z Disabled control of system clock
I20240503 13:55:40.031816 6537 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
/tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.98.126:44199
--webserver_interface=127.6.98.126
--webserver_port=0
--builtin_ntp_servers=127.6.98.84:44499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.98.126:44199
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240503 13:55:40.139225 7409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:55:40.139469 7409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:55:40.139523 7409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:55:40.142594 7409 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 13:55:40.142668 7409 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 13:55:40.142694 7409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:55:40.142724 7409 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 13:55:40.142755 7409 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 13:55:40.146386 7409 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.98.84:44499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.6.98.126:44199
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.98.126:44199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.6.98.126
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.7409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:55:40.147290 7409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:55:40.148438 7409 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:55:40.155368 7415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:55:40.155601 7417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:55:40.155858 7414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:55:40.156447 7409 server_base.cc:1034] running on GCE node
I20240503 13:55:40.157013 7409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:55:40.158016 7409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:55:40.159195 7409 hybrid_clock.cc:648] HybridClock initialized: now 1714744540159174 us; error 41 us; skew 500 ppm
I20240503 13:55:40.161260 7409 webserver.cc:460] Webserver started at http://127.6.98.126:41575/ using document root <none> and password file <none>
I20240503 13:55:40.161806 7409 fs_manager.cc:359] Metadata directory not provided
I20240503 13:55:40.161865 7409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:55:40.162045 7409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:55:40.163678 7409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "8090243dfe8247b58111b00608e8eb25"
format_stamp: "Formatted at 2024-05-03 13:55:40 on dist-test-slave-zg9z"
I20240503 13:55:40.164129 7409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "8090243dfe8247b58111b00608e8eb25"
format_stamp: "Formatted at 2024-05-03 13:55:40 on dist-test-slave-zg9z"
I20240503 13:55:40.167423 7409 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 13:55:40.169617 7423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:55:40.170596 7409 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 13:55:40.170704 7409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/master-0/wal
uuid: "8090243dfe8247b58111b00608e8eb25"
format_stamp: "Formatted at 2024-05-03 13:55:40 on dist-test-slave-zg9z"
I20240503 13:55:40.170789 7409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:55:40.185999 7409 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:55:40.186658 7409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:55:40.186812 7409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:55:40.193413 7409 rpc_server.cc:307] RPC server started. Bound to: 127.6.98.126:44199
I20240503 13:55:40.193450 7475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.98.126:44199 every 8 connection(s)
I20240503 13:55:40.194356 7409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 13:55:40.196768 6537 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu as pid 7409
I20240503 13:55:40.196882 6537 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 13:55:40.197055 7476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:55:40.203532 7476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8090243dfe8247b58111b00608e8eb25: Bootstrap starting.
I20240503 13:55:40.205982 7476 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8090243dfe8247b58111b00608e8eb25: Neither blocks nor log segments found. Creating new log.
I20240503 13:55:40.206643 7476 log.cc:824] T 00000000000000000000000000000000 P 8090243dfe8247b58111b00608e8eb25: Log is configured to *not* fsync() on all Append() calls
I20240503 13:55:40.208365 7476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8090243dfe8247b58111b00608e8eb25: No bootstrap required, opened a new log
I20240503 13:55:40.210880 7476 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8090243dfe8247b58111b00608e8eb25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8090243dfe8247b58111b00608e8eb25" member_type: VOTER last_known_addr { host: "127.6.98.126" port: 44199 } }
I20240503 13:55:40.211069 7476 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8090243dfe8247b58111b00608e8eb25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:55:40.211103 7476 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8090243dfe8247b58111b00608e8eb25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8090243dfe8247b58111b00608e8eb25, State: Initialized, Role: FOLLOWER
I20240503 13:55:40.211527 7476 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8090243dfe8247b58111b00608e8eb25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8090243dfe8247b58111b00608e8eb25" member_type: VOTER last_known_addr { host: "127.6.98.126" port: 44199 } }
I20240503 13:55:40.211686 7476 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8090243dfe8247b58111b00608e8eb25 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:55:40.211755 7476 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8090243dfe8247b58111b00608e8eb25 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:55:40.211864 7476 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8090243dfe8247b58111b00608e8eb25 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:55:40.212718 7476 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8090243dfe8247b58111b00608e8eb25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8090243dfe8247b58111b00608e8eb25" member_type: VOTER last_known_addr { host: "127.6.98.126" port: 44199 } }
I20240503 13:55:40.213038 7476 leader_election.cc:302] T 00000000000000000000000000000000 P 8090243dfe8247b58111b00608e8eb25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8090243dfe8247b58111b00608e8eb25; no voters:
I20240503 13:55:40.213315 7476 leader_election.cc:288] T 00000000000000000000000000000000 P 8090243dfe8247b58111b00608e8eb25 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 13:55:40.213438 7481 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8090243dfe8247b58111b00608e8eb25 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:55:40.213665 7481 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8090243dfe8247b58111b00608e8eb25 [term 1 LEADER]: Becoming Leader. State: Replica: 8090243dfe8247b58111b00608e8eb25, State: Running, Role: LEADER
I20240503 13:55:40.213999 7481 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8090243dfe8247b58111b00608e8eb25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8090243dfe8247b58111b00608e8eb25" member_type: VOTER last_known_addr { host: "127.6.98.126" port: 44199 } }
I20240503 13:55:40.214346 7476 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8090243dfe8247b58111b00608e8eb25 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 13:55:40.215579 7483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8090243dfe8247b58111b00608e8eb25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8090243dfe8247b58111b00608e8eb25. Latest consensus state: current_term: 1 leader_uuid: "8090243dfe8247b58111b00608e8eb25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8090243dfe8247b58111b00608e8eb25" member_type: VOTER last_known_addr { host: "127.6.98.126" port: 44199 } } }
I20240503 13:55:40.215703 7483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8090243dfe8247b58111b00608e8eb25 [sys.catalog]: This master's current role is: LEADER
I20240503 13:55:40.215896 7482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8090243dfe8247b58111b00608e8eb25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8090243dfe8247b58111b00608e8eb25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8090243dfe8247b58111b00608e8eb25" member_type: VOTER last_known_addr { host: "127.6.98.126" port: 44199 } } }
I20240503 13:55:40.215996 7482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8090243dfe8247b58111b00608e8eb25 [sys.catalog]: This master's current role is: LEADER
I20240503 13:55:40.216015 7490 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 13:55:40.219035 7490 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 13:55:40.224712 7490 catalog_manager.cc:1347] Generated new cluster ID: 06551c27993f4efab8ab58ece38c9457
I20240503 13:55:40.224785 7490 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 13:55:40.238894 7490 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 13:55:40.239814 7490 catalog_manager.cc:1504] Loading token signing keys...
I20240503 13:55:40.249543 7490 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 8090243dfe8247b58111b00608e8eb25: Generated new TSK 0
I20240503 13:55:40.250383 7490 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 13:55:40.264057 6537 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
/tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.98.65:0
--local_ip_for_outbound_sockets=127.6.98.65
--webserver_interface=127.6.98.65
--webserver_port=0
--tserver_master_addrs=127.6.98.126:44199
--builtin_ntp_servers=127.6.98.84:44499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 13:55:40.370642 7500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:55:40.370874 7500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:55:40.370945 7500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:55:40.374070 7500 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 13:55:40.374136 7500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:55:40.374220 7500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.98.65
I20240503 13:55:40.377843 7500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.98.84:44499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.98.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.6.98.65
--webserver_port=0
--tserver_master_addrs=127.6.98.126:44199
--never_fsync=true
--heap_profile_path=/tmp/kudu.7500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.98.65
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:55:40.378728 7500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:55:40.379858 7500 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:55:40.388218 7505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:55:40.388231 7506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:55:40.388507 7508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:55:40.389046 7500 server_base.cc:1034] running on GCE node
I20240503 13:55:40.389448 7500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:55:40.390029 7500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:55:40.391172 7500 hybrid_clock.cc:648] HybridClock initialized: now 1714744540391152 us; error 36 us; skew 500 ppm
I20240503 13:55:40.393128 7500 webserver.cc:460] Webserver started at http://127.6.98.65:39095/ using document root <none> and password file <none>
I20240503 13:55:40.393661 7500 fs_manager.cc:359] Metadata directory not provided
I20240503 13:55:40.393715 7500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:55:40.393886 7500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:55:40.395522 7500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "83e34b5317b84676b7852268d7eaa6e5"
format_stamp: "Formatted at 2024-05-03 13:55:40 on dist-test-slave-zg9z"
I20240503 13:55:40.395995 7500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "83e34b5317b84676b7852268d7eaa6e5"
format_stamp: "Formatted at 2024-05-03 13:55:40 on dist-test-slave-zg9z"
I20240503 13:55:40.399312 7500 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 13:55:40.401687 7514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:55:40.402789 7500 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 13:55:40.402912 7500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "83e34b5317b84676b7852268d7eaa6e5"
format_stamp: "Formatted at 2024-05-03 13:55:40 on dist-test-slave-zg9z"
I20240503 13:55:40.403015 7500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:55:40.423689 7500 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:55:40.424387 7500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:55:40.424614 7500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:55:40.425199 7500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:55:40.426117 7500 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:55:40.426167 7500 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:55:40.426221 7500 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:55:40.426246 7500 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:55:40.434392 7500 rpc_server.cc:307] RPC server started. Bound to: 127.6.98.65:34785
I20240503 13:55:40.434434 7629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.98.65:34785 every 8 connection(s)
I20240503 13:55:40.435354 7500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 13:55:40.440066 6537 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu as pid 7500
I20240503 13:55:40.440192 6537 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 13:55:40.442598 6537 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
/tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.98.66:0
--local_ip_for_outbound_sockets=127.6.98.66
--webserver_interface=127.6.98.66
--webserver_port=0
--tserver_master_addrs=127.6.98.126:44199
--builtin_ntp_servers=127.6.98.84:44499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 13:55:40.445114 7630 heartbeater.cc:347] Connected to a master server at 127.6.98.126:44199
I20240503 13:55:40.445345 7630 heartbeater.cc:464] Registering TS with master...
I20240503 13:55:40.445811 7630 heartbeater.cc:511] Master 127.6.98.126:44199 requested a full tablet report, sending...
I20240503 13:55:40.447270 7440 ts_manager.cc:194] Registered new tserver with Master: 83e34b5317b84676b7852268d7eaa6e5 (127.6.98.65:34785)
I20240503 13:55:40.448550 7440 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.98.65:57865
W20240503 13:55:40.550153 7633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:55:40.550402 7633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:55:40.550464 7633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:55:40.553556 7633 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 13:55:40.553632 7633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:55:40.553722 7633 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.98.66
I20240503 13:55:40.557454 7633 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.98.84:44499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.98.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.6.98.66
--webserver_port=0
--tserver_master_addrs=127.6.98.126:44199
--never_fsync=true
--heap_profile_path=/tmp/kudu.7633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.98.66
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:55:40.558434 7633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:55:40.559590 7633 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 13:55:40.568574 7633 server_base.cc:1034] running on GCE node
W20240503 13:55:40.568806 7638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:55:40.568960 7641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:55:40.568943 7639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:55:40.569510 7633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:55:40.570104 7633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:55:40.571256 7633 hybrid_clock.cc:648] HybridClock initialized: now 1714744540571226 us; error 47 us; skew 500 ppm
I20240503 13:55:40.573494 7633 webserver.cc:460] Webserver started at http://127.6.98.66:36703/ using document root <none> and password file <none>
I20240503 13:55:40.574111 7633 fs_manager.cc:359] Metadata directory not provided
I20240503 13:55:40.574175 7633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:55:40.574352 7633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:55:40.576082 7633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "d158c837291346ca9f2dd216f91a8572"
format_stamp: "Formatted at 2024-05-03 13:55:40 on dist-test-slave-zg9z"
I20240503 13:55:40.576586 7633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "d158c837291346ca9f2dd216f91a8572"
format_stamp: "Formatted at 2024-05-03 13:55:40 on dist-test-slave-zg9z"
I20240503 13:55:40.579977 7633 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20240503 13:55:40.582186 7647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:55:40.583186 7633 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 13:55:40.583292 7633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "d158c837291346ca9f2dd216f91a8572"
format_stamp: "Formatted at 2024-05-03 13:55:40 on dist-test-slave-zg9z"
I20240503 13:55:40.583376 7633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:55:40.600694 7633 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:55:40.601347 7633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:55:40.601519 7633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:55:40.602069 7633 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:55:40.603001 7633 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:55:40.603055 7633 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:55:40.603113 7633 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:55:40.603152 7633 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:55:40.612247 7633 rpc_server.cc:307] RPC server started. Bound to: 127.6.98.66:33311
I20240503 13:55:40.612272 7761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.98.66:33311 every 8 connection(s)
I20240503 13:55:40.613220 7633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 13:55:40.618868 6537 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu as pid 7633
I20240503 13:55:40.618960 6537 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 13:55:40.621084 6537 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
/tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.98.67:0
--local_ip_for_outbound_sockets=127.6.98.67
--webserver_interface=127.6.98.67
--webserver_port=0
--tserver_master_addrs=127.6.98.126:44199
--builtin_ntp_servers=127.6.98.84:44499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 13:55:40.622100 7762 heartbeater.cc:347] Connected to a master server at 127.6.98.126:44199
I20240503 13:55:40.622340 7762 heartbeater.cc:464] Registering TS with master...
I20240503 13:55:40.622860 7762 heartbeater.cc:511] Master 127.6.98.126:44199 requested a full tablet report, sending...
I20240503 13:55:40.623811 7440 ts_manager.cc:194] Registered new tserver with Master: d158c837291346ca9f2dd216f91a8572 (127.6.98.66:33311)
I20240503 13:55:40.624541 7440 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.98.66:52939
W20240503 13:55:40.726711 7766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:55:40.726938 7766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:55:40.726987 7766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:55:40.730026 7766 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 13:55:40.730095 7766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:55:40.730176 7766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.98.67
I20240503 13:55:40.733860 7766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.98.84:44499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.98.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.6.98.67
--webserver_port=0
--tserver_master_addrs=127.6.98.126:44199
--never_fsync=true
--heap_profile_path=/tmp/kudu.7766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.98.67
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:55:40.734798 7766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:55:40.735954 7766 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:55:40.744221 7771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:55:40.744688 7774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:55:40.744782 7772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:55:40.745254 7766 server_base.cc:1034] running on GCE node
I20240503 13:55:40.745656 7766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:55:40.746229 7766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:55:40.747411 7766 hybrid_clock.cc:648] HybridClock initialized: now 1714744540747393 us; error 45 us; skew 500 ppm
I20240503 13:55:40.749506 7766 webserver.cc:460] Webserver started at http://127.6.98.67:40599/ using document root <none> and password file <none>
I20240503 13:55:40.750052 7766 fs_manager.cc:359] Metadata directory not provided
I20240503 13:55:40.750113 7766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:55:40.750298 7766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:55:40.751982 7766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "06b84a83a711445f9071c2408ce2ca72"
format_stamp: "Formatted at 2024-05-03 13:55:40 on dist-test-slave-zg9z"
I20240503 13:55:40.752439 7766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "06b84a83a711445f9071c2408ce2ca72"
format_stamp: "Formatted at 2024-05-03 13:55:40 on dist-test-slave-zg9z"
I20240503 13:55:40.755813 7766 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20240503 13:55:40.758145 7780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:55:40.759333 7766 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20240503 13:55:40.759443 7766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "06b84a83a711445f9071c2408ce2ca72"
format_stamp: "Formatted at 2024-05-03 13:55:40 on dist-test-slave-zg9z"
I20240503 13:55:40.759532 7766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:55:40.790494 7766 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:55:40.791108 7766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:55:40.791260 7766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:55:40.791797 7766 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:55:40.792721 7766 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:55:40.792769 7766 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:55:40.792824 7766 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:55:40.792847 7766 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:55:40.800972 7766 rpc_server.cc:307] RPC server started. Bound to: 127.6.98.67:41181
I20240503 13:55:40.800994 7894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.98.67:41181 every 8 connection(s)
I20240503 13:55:40.801918 7766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 13:55:40.807482 6537 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu as pid 7766
I20240503 13:55:40.807590 6537 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 13:55:40.809938 6537 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
/tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.98.68:0
--local_ip_for_outbound_sockets=127.6.98.68
--webserver_interface=127.6.98.68
--webserver_port=0
--tserver_master_addrs=127.6.98.126:44199
--builtin_ntp_servers=127.6.98.84:44499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 13:55:40.811255 7895 heartbeater.cc:347] Connected to a master server at 127.6.98.126:44199
I20240503 13:55:40.811527 7895 heartbeater.cc:464] Registering TS with master...
I20240503 13:55:40.812074 7895 heartbeater.cc:511] Master 127.6.98.126:44199 requested a full tablet report, sending...
I20240503 13:55:40.813222 7440 ts_manager.cc:194] Registered new tserver with Master: 06b84a83a711445f9071c2408ce2ca72 (127.6.98.67:41181)
I20240503 13:55:40.813978 7440 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.98.67:53755
W20240503 13:55:40.917295 7898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:55:40.917538 7898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:55:40.917584 7898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:55:40.920637 7898 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 13:55:40.920706 7898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:55:40.920773 7898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.98.68
I20240503 13:55:40.924440 7898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.98.84:44499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.98.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.6.98.68
--webserver_port=0
--tserver_master_addrs=127.6.98.126:44199
--never_fsync=true
--heap_profile_path=/tmp/kudu.7898
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.98.68
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:55:40.925446 7898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:55:40.926570 7898 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:55:40.934429 7906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:55:40.934774 7903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:55:40.935253 7904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:55:40.935720 7898 server_base.cc:1034] running on GCE node
I20240503 13:55:40.936133 7898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:55:40.936786 7898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:55:40.937935 7898 hybrid_clock.cc:648] HybridClock initialized: now 1714744540937922 us; error 28 us; skew 500 ppm
I20240503 13:55:40.939882 7898 webserver.cc:460] Webserver started at http://127.6.98.68:46471/ using document root <none> and password file <none>
I20240503 13:55:40.940439 7898 fs_manager.cc:359] Metadata directory not provided
I20240503 13:55:40.940522 7898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:55:40.940711 7898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:55:40.942373 7898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "33c9030181f64b7bba94366dbb3ef7d8"
format_stamp: "Formatted at 2024-05-03 13:55:40 on dist-test-slave-zg9z"
I20240503 13:55:40.942819 7898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "33c9030181f64b7bba94366dbb3ef7d8"
format_stamp: "Formatted at 2024-05-03 13:55:40 on dist-test-slave-zg9z"
I20240503 13:55:40.946262 7898 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 13:55:40.948549 7912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:55:40.949635 7898 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 13:55:40.949743 7898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "33c9030181f64b7bba94366dbb3ef7d8"
format_stamp: "Formatted at 2024-05-03 13:55:40 on dist-test-slave-zg9z"
I20240503 13:55:40.949851 7898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:55:40.962471 7898 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:55:40.963102 7898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:55:40.963253 7898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:55:40.963769 7898 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:55:40.964731 7898 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:55:40.964782 7898 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:55:40.964828 7898 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:55:40.964855 7898 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:55:40.972883 7898 rpc_server.cc:307] RPC server started. Bound to: 127.6.98.68:34095
I20240503 13:55:40.972923 8025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.98.68:34095 every 8 connection(s)
I20240503 13:55:40.973883 7898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 13:55:40.976183 6537 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu as pid 7898
I20240503 13:55:40.976281 6537 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 13:55:40.978284 6537 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
/tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.98.69:0
--local_ip_for_outbound_sockets=127.6.98.69
--webserver_interface=127.6.98.69
--webserver_port=0
--tserver_master_addrs=127.6.98.126:44199
--builtin_ntp_servers=127.6.98.84:44499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 13:55:40.982905 8026 heartbeater.cc:347] Connected to a master server at 127.6.98.126:44199
I20240503 13:55:40.983147 8026 heartbeater.cc:464] Registering TS with master...
I20240503 13:55:40.983605 8026 heartbeater.cc:511] Master 127.6.98.126:44199 requested a full tablet report, sending...
I20240503 13:55:40.984567 7440 ts_manager.cc:194] Registered new tserver with Master: 33c9030181f64b7bba94366dbb3ef7d8 (127.6.98.68:34095)
I20240503 13:55:40.985111 7440 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.98.68:33227
W20240503 13:55:41.085920 8029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:55:41.086154 8029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:55:41.086215 8029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:55:41.089293 8029 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 13:55:41.089365 8029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:55:41.089453 8029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.98.69
I20240503 13:55:41.093282 8029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.98.84:44499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.98.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.6.98.69
--webserver_port=0
--tserver_master_addrs=127.6.98.126:44199
--never_fsync=true
--heap_profile_path=/tmp/kudu.8029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.98.69
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:55:41.094213 8029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:55:41.095348 8029 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:55:41.103708 8034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:55:41.103750 8035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:55:41.104203 8037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:55:41.104976 8029 server_base.cc:1034] running on GCE node
I20240503 13:55:41.105369 8029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:55:41.105911 8029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:55:41.107061 8029 hybrid_clock.cc:648] HybridClock initialized: now 1714744541107044 us; error 34 us; skew 500 ppm
I20240503 13:55:41.109123 8029 webserver.cc:460] Webserver started at http://127.6.98.69:33237/ using document root <none> and password file <none>
I20240503 13:55:41.109663 8029 fs_manager.cc:359] Metadata directory not provided
I20240503 13:55:41.109715 8029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:55:41.109874 8029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:55:41.111554 8029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "32cbeb4ffe414abc811ba64fe0c650ad"
format_stamp: "Formatted at 2024-05-03 13:55:41 on dist-test-slave-zg9z"
I20240503 13:55:41.111999 8029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "32cbeb4ffe414abc811ba64fe0c650ad"
format_stamp: "Formatted at 2024-05-03 13:55:41 on dist-test-slave-zg9z"
I20240503 13:55:41.115501 8029 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20240503 13:55:41.117746 8044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:55:41.118784 8029 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 13:55:41.118891 8029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "32cbeb4ffe414abc811ba64fe0c650ad"
format_stamp: "Formatted at 2024-05-03 13:55:41 on dist-test-slave-zg9z"
I20240503 13:55:41.118989 8029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:55:41.134997 8029 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:55:41.135577 8029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:55:41.135720 8029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:55:41.136340 8029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:55:41.137255 8029 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:55:41.137301 8029 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:55:41.137343 8029 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:55:41.137372 8029 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:55:41.145534 8029 rpc_server.cc:307] RPC server started. Bound to: 127.6.98.69:40881
I20240503 13:55:41.145558 8157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.98.69:40881 every 8 connection(s)
I20240503 13:55:41.146412 8029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 13:55:41.153755 6537 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu as pid 8029
I20240503 13:55:41.153880 6537 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 13:55:41.154820 8158 heartbeater.cc:347] Connected to a master server at 127.6.98.126:44199
I20240503 13:55:41.155042 8158 heartbeater.cc:464] Registering TS with master...
I20240503 13:55:41.155567 8158 heartbeater.cc:511] Master 127.6.98.126:44199 requested a full tablet report, sending...
I20240503 13:55:41.156431 7440 ts_manager.cc:194] Registered new tserver with Master: 32cbeb4ffe414abc811ba64fe0c650ad (127.6.98.69:40881)
I20240503 13:55:41.156993 7440 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.98.69:47813
I20240503 13:55:41.167866 6537 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 13:55:41.176023 7440 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35802:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 13:55:41.193672 7960 tablet_service.cc:1467] Processing CreateTablet for tablet 357e0dcea1334d388e4cfcc931fda43a (DEFAULT_TABLE table=TestTable [id=0d52bd0b89f44f5699fe113df65bf1b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 13:55:41.194061 7828 tablet_service.cc:1467] Processing CreateTablet for tablet 357e0dcea1334d388e4cfcc931fda43a (DEFAULT_TABLE table=TestTable [id=0d52bd0b89f44f5699fe113df65bf1b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 13:55:41.194061 8092 tablet_service.cc:1467] Processing CreateTablet for tablet 357e0dcea1334d388e4cfcc931fda43a (DEFAULT_TABLE table=TestTable [id=0d52bd0b89f44f5699fe113df65bf1b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 13:55:41.194583 7960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 357e0dcea1334d388e4cfcc931fda43a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:55:41.194940 7828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 357e0dcea1334d388e4cfcc931fda43a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:55:41.194939 8092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 357e0dcea1334d388e4cfcc931fda43a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:55:41.200970 8175 tablet_bootstrap.cc:492] T 357e0dcea1334d388e4cfcc931fda43a P 33c9030181f64b7bba94366dbb3ef7d8: Bootstrap starting.
I20240503 13:55:41.201164 8177 tablet_bootstrap.cc:492] T 357e0dcea1334d388e4cfcc931fda43a P 06b84a83a711445f9071c2408ce2ca72: Bootstrap starting.
I20240503 13:55:41.201535 8176 tablet_bootstrap.cc:492] T 357e0dcea1334d388e4cfcc931fda43a P 32cbeb4ffe414abc811ba64fe0c650ad: Bootstrap starting.
I20240503 13:55:41.203065 8175 tablet_bootstrap.cc:654] T 357e0dcea1334d388e4cfcc931fda43a P 33c9030181f64b7bba94366dbb3ef7d8: Neither blocks nor log segments found. Creating new log.
I20240503 13:55:41.203194 8177 tablet_bootstrap.cc:654] T 357e0dcea1334d388e4cfcc931fda43a P 06b84a83a711445f9071c2408ce2ca72: Neither blocks nor log segments found. Creating new log.
I20240503 13:55:41.203806 8177 log.cc:824] T 357e0dcea1334d388e4cfcc931fda43a P 06b84a83a711445f9071c2408ce2ca72: Log is configured to *not* fsync() on all Append() calls
I20240503 13:55:41.203838 8175 log.cc:824] T 357e0dcea1334d388e4cfcc931fda43a P 33c9030181f64b7bba94366dbb3ef7d8: Log is configured to *not* fsync() on all Append() calls
I20240503 13:55:41.205446 8175 tablet_bootstrap.cc:492] T 357e0dcea1334d388e4cfcc931fda43a P 33c9030181f64b7bba94366dbb3ef7d8: No bootstrap required, opened a new log
I20240503 13:55:41.205608 8175 ts_tablet_manager.cc:1367] T 357e0dcea1334d388e4cfcc931fda43a P 33c9030181f64b7bba94366dbb3ef7d8: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 13:55:41.205788 8176 tablet_bootstrap.cc:654] T 357e0dcea1334d388e4cfcc931fda43a P 32cbeb4ffe414abc811ba64fe0c650ad: Neither blocks nor log segments found. Creating new log.
I20240503 13:55:41.206574 8176 log.cc:824] T 357e0dcea1334d388e4cfcc931fda43a P 32cbeb4ffe414abc811ba64fe0c650ad: Log is configured to *not* fsync() on all Append() calls
I20240503 13:55:41.206650 8177 tablet_bootstrap.cc:492] T 357e0dcea1334d388e4cfcc931fda43a P 06b84a83a711445f9071c2408ce2ca72: No bootstrap required, opened a new log
I20240503 13:55:41.207679 8177 ts_tablet_manager.cc:1367] T 357e0dcea1334d388e4cfcc931fda43a P 06b84a83a711445f9071c2408ce2ca72: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240503 13:55:41.208127 8176 tablet_bootstrap.cc:492] T 357e0dcea1334d388e4cfcc931fda43a P 32cbeb4ffe414abc811ba64fe0c650ad: No bootstrap required, opened a new log
I20240503 13:55:41.208268 8176 ts_tablet_manager.cc:1367] T 357e0dcea1334d388e4cfcc931fda43a P 32cbeb4ffe414abc811ba64fe0c650ad: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240503 13:55:41.209146 8175 raft_consensus.cc:348] T 357e0dcea1334d388e4cfcc931fda43a P 33c9030181f64b7bba94366dbb3ef7d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c9030181f64b7bba94366dbb3ef7d8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 34095 } } peers { permanent_uuid: "06b84a83a711445f9071c2408ce2ca72" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 41181 } } peers { permanent_uuid: "32cbeb4ffe414abc811ba64fe0c650ad" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 40881 } }
I20240503 13:55:41.209350 8175 raft_consensus.cc:374] T 357e0dcea1334d388e4cfcc931fda43a P 33c9030181f64b7bba94366dbb3ef7d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:55:41.209396 8175 raft_consensus.cc:729] T 357e0dcea1334d388e4cfcc931fda43a P 33c9030181f64b7bba94366dbb3ef7d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33c9030181f64b7bba94366dbb3ef7d8, State: Initialized, Role: FOLLOWER
I20240503 13:55:41.210042 8175 consensus_queue.cc:260] T 357e0dcea1334d388e4cfcc931fda43a P 33c9030181f64b7bba94366dbb3ef7d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c9030181f64b7bba94366dbb3ef7d8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 34095 } } peers { permanent_uuid: "06b84a83a711445f9071c2408ce2ca72" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 41181 } } peers { permanent_uuid: "32cbeb4ffe414abc811ba64fe0c650ad" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 40881 } }
I20240503 13:55:41.210937 8175 ts_tablet_manager.cc:1398] T 357e0dcea1334d388e4cfcc931fda43a P 33c9030181f64b7bba94366dbb3ef7d8: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
I20240503 13:55:41.211335 8026 heartbeater.cc:503] Master 127.6.98.126:44199 was elected leader, sending a full tablet report...
I20240503 13:55:41.211336 8177 raft_consensus.cc:348] T 357e0dcea1334d388e4cfcc931fda43a P 06b84a83a711445f9071c2408ce2ca72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c9030181f64b7bba94366dbb3ef7d8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 34095 } } peers { permanent_uuid: "06b84a83a711445f9071c2408ce2ca72" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 41181 } } peers { permanent_uuid: "32cbeb4ffe414abc811ba64fe0c650ad" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 40881 } }
I20240503 13:55:41.211536 8177 raft_consensus.cc:374] T 357e0dcea1334d388e4cfcc931fda43a P 06b84a83a711445f9071c2408ce2ca72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:55:41.211580 8177 raft_consensus.cc:729] T 357e0dcea1334d388e4cfcc931fda43a P 06b84a83a711445f9071c2408ce2ca72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06b84a83a711445f9071c2408ce2ca72, State: Initialized, Role: FOLLOWER
I20240503 13:55:41.211683 8176 raft_consensus.cc:348] T 357e0dcea1334d388e4cfcc931fda43a P 32cbeb4ffe414abc811ba64fe0c650ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c9030181f64b7bba94366dbb3ef7d8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 34095 } } peers { permanent_uuid: "06b84a83a711445f9071c2408ce2ca72" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 41181 } } peers { permanent_uuid: "32cbeb4ffe414abc811ba64fe0c650ad" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 40881 } }
I20240503 13:55:41.211838 8176 raft_consensus.cc:374] T 357e0dcea1334d388e4cfcc931fda43a P 32cbeb4ffe414abc811ba64fe0c650ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:55:41.211887 8176 raft_consensus.cc:729] T 357e0dcea1334d388e4cfcc931fda43a P 32cbeb4ffe414abc811ba64fe0c650ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32cbeb4ffe414abc811ba64fe0c650ad, State: Initialized, Role: FOLLOWER
I20240503 13:55:41.211977 8177 consensus_queue.cc:260] T 357e0dcea1334d388e4cfcc931fda43a P 06b84a83a711445f9071c2408ce2ca72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c9030181f64b7bba94366dbb3ef7d8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 34095 } } peers { permanent_uuid: "06b84a83a711445f9071c2408ce2ca72" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 41181 } } peers { permanent_uuid: "32cbeb4ffe414abc811ba64fe0c650ad" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 40881 } }
I20240503 13:55:41.212241 8176 consensus_queue.cc:260] T 357e0dcea1334d388e4cfcc931fda43a P 32cbeb4ffe414abc811ba64fe0c650ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c9030181f64b7bba94366dbb3ef7d8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 34095 } } peers { permanent_uuid: "06b84a83a711445f9071c2408ce2ca72" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 41181 } } peers { permanent_uuid: "32cbeb4ffe414abc811ba64fe0c650ad" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 40881 } }
I20240503 13:55:41.212787 8177 ts_tablet_manager.cc:1398] T 357e0dcea1334d388e4cfcc931fda43a P 06b84a83a711445f9071c2408ce2ca72: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240503 13:55:41.212883 7895 heartbeater.cc:503] Master 127.6.98.126:44199 was elected leader, sending a full tablet report...
I20240503 13:55:41.213352 8158 heartbeater.cc:503] Master 127.6.98.126:44199 was elected leader, sending a full tablet report...
I20240503 13:55:41.213883 8176 ts_tablet_manager.cc:1398] T 357e0dcea1334d388e4cfcc931fda43a P 32cbeb4ffe414abc811ba64fe0c650ad: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20240503 13:55:41.224717 8027 tablet.cc:2369] T 357e0dcea1334d388e4cfcc931fda43a P 33c9030181f64b7bba94366dbb3ef7d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 13:55:41.258172 8184 raft_consensus.cc:482] T 357e0dcea1334d388e4cfcc931fda43a P 32cbeb4ffe414abc811ba64fe0c650ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:55:41.258374 8184 raft_consensus.cc:504] T 357e0dcea1334d388e4cfcc931fda43a P 32cbeb4ffe414abc811ba64fe0c650ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c9030181f64b7bba94366dbb3ef7d8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 34095 } } peers { permanent_uuid: "06b84a83a711445f9071c2408ce2ca72" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 41181 } } peers { permanent_uuid: "32cbeb4ffe414abc811ba64fe0c650ad" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 40881 } }
I20240503 13:55:41.259418 8184 leader_election.cc:288] T 357e0dcea1334d388e4cfcc931fda43a P 32cbeb4ffe414abc811ba64fe0c650ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33c9030181f64b7bba94366dbb3ef7d8 (127.6.98.68:34095), 06b84a83a711445f9071c2408ce2ca72 (127.6.98.67:41181)
I20240503 13:55:41.263442 7848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "357e0dcea1334d388e4cfcc931fda43a" candidate_uuid: "32cbeb4ffe414abc811ba64fe0c650ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06b84a83a711445f9071c2408ce2ca72" is_pre_election: true
I20240503 13:55:41.263535 7980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "357e0dcea1334d388e4cfcc931fda43a" candidate_uuid: "32cbeb4ffe414abc811ba64fe0c650ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33c9030181f64b7bba94366dbb3ef7d8" is_pre_election: true
I20240503 13:55:41.263746 7848 raft_consensus.cc:2454] T 357e0dcea1334d388e4cfcc931fda43a P 06b84a83a711445f9071c2408ce2ca72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32cbeb4ffe414abc811ba64fe0c650ad in term 0.
I20240503 13:55:41.263811 7980 raft_consensus.cc:2454] T 357e0dcea1334d388e4cfcc931fda43a P 33c9030181f64b7bba94366dbb3ef7d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32cbeb4ffe414abc811ba64fe0c650ad in term 0.
I20240503 13:55:41.264235 8045 leader_election.cc:302] T 357e0dcea1334d388e4cfcc931fda43a P 32cbeb4ffe414abc811ba64fe0c650ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06b84a83a711445f9071c2408ce2ca72, 32cbeb4ffe414abc811ba64fe0c650ad; no voters:
I20240503 13:55:41.264499 8184 raft_consensus.cc:2781] T 357e0dcea1334d388e4cfcc931fda43a P 32cbeb4ffe414abc811ba64fe0c650ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:55:41.264586 8184 raft_consensus.cc:482] T 357e0dcea1334d388e4cfcc931fda43a P 32cbeb4ffe414abc811ba64fe0c650ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:55:41.264636 8184 raft_consensus.cc:3037] T 357e0dcea1334d388e4cfcc931fda43a P 32cbeb4ffe414abc811ba64fe0c650ad [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:55:41.265493 8184 raft_consensus.cc:504] T 357e0dcea1334d388e4cfcc931fda43a P 32cbeb4ffe414abc811ba64fe0c650ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c9030181f64b7bba94366dbb3ef7d8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 34095 } } peers { permanent_uuid: "06b84a83a711445f9071c2408ce2ca72" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 41181 } } peers { permanent_uuid: "32cbeb4ffe414abc811ba64fe0c650ad" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 40881 } }
I20240503 13:55:41.265827 8184 leader_election.cc:288] T 357e0dcea1334d388e4cfcc931fda43a P 32cbeb4ffe414abc811ba64fe0c650ad [CANDIDATE]: Term 1 election: Requested vote from peers 33c9030181f64b7bba94366dbb3ef7d8 (127.6.98.68:34095), 06b84a83a711445f9071c2408ce2ca72 (127.6.98.67:41181)
I20240503 13:55:41.266137 7848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "357e0dcea1334d388e4cfcc931fda43a" candidate_uuid: "32cbeb4ffe414abc811ba64fe0c650ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06b84a83a711445f9071c2408ce2ca72"
I20240503 13:55:41.266157 7980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "357e0dcea1334d388e4cfcc931fda43a" candidate_uuid: "32cbeb4ffe414abc811ba64fe0c650ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33c9030181f64b7bba94366dbb3ef7d8"
I20240503 13:55:41.266248 7980 raft_consensus.cc:3037] T 357e0dcea1334d388e4cfcc931fda43a P 33c9030181f64b7bba94366dbb3ef7d8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:55:41.266252 7848 raft_consensus.cc:3037] T 357e0dcea1334d388e4cfcc931fda43a P 06b84a83a711445f9071c2408ce2ca72 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:55:41.267068 7980 raft_consensus.cc:2454] T 357e0dcea1334d388e4cfcc931fda43a P 33c9030181f64b7bba94366dbb3ef7d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32cbeb4ffe414abc811ba64fe0c650ad in term 1.
I20240503 13:55:41.267081 7848 raft_consensus.cc:2454] T 357e0dcea1334d388e4cfcc931fda43a P 06b84a83a711445f9071c2408ce2ca72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32cbeb4ffe414abc811ba64fe0c650ad in term 1.
I20240503 13:55:41.267369 8046 leader_election.cc:302] T 357e0dcea1334d388e4cfcc931fda43a P 32cbeb4ffe414abc811ba64fe0c650ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32cbeb4ffe414abc811ba64fe0c650ad, 33c9030181f64b7bba94366dbb3ef7d8; no voters:
I20240503 13:55:41.267537 8184 raft_consensus.cc:2781] T 357e0dcea1334d388e4cfcc931fda43a P 32cbeb4ffe414abc811ba64fe0c650ad [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:55:41.267755 8184 raft_consensus.cc:686] T 357e0dcea1334d388e4cfcc931fda43a P 32cbeb4ffe414abc811ba64fe0c650ad [term 1 LEADER]: Becoming Leader. State: Replica: 32cbeb4ffe414abc811ba64fe0c650ad, State: Running, Role: LEADER
I20240503 13:55:41.268105 8184 consensus_queue.cc:237] T 357e0dcea1334d388e4cfcc931fda43a P 32cbeb4ffe414abc811ba64fe0c650ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c9030181f64b7bba94366dbb3ef7d8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 34095 } } peers { permanent_uuid: "06b84a83a711445f9071c2408ce2ca72" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 41181 } } peers { permanent_uuid: "32cbeb4ffe414abc811ba64fe0c650ad" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 40881 } }
I20240503 13:55:41.270995 7440 catalog_manager.cc:5484] T 357e0dcea1334d388e4cfcc931fda43a P 32cbeb4ffe414abc811ba64fe0c650ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 32cbeb4ffe414abc811ba64fe0c650ad (127.6.98.69). New cstate: current_term: 1 leader_uuid: "32cbeb4ffe414abc811ba64fe0c650ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c9030181f64b7bba94366dbb3ef7d8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 34095 } } peers { permanent_uuid: "06b84a83a711445f9071c2408ce2ca72" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 41181 } } peers { permanent_uuid: "32cbeb4ffe414abc811ba64fe0c650ad" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 40881 } } }
I20240503 13:55:41.285641 6537 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 13:55:41.287132 6537 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 83e34b5317b84676b7852268d7eaa6e5 to finish bootstrapping
I20240503 13:55:41.293213 6537 ts_itest-base.cc:233] Waiting for 0 tablets on tserver d158c837291346ca9f2dd216f91a8572 to finish bootstrapping
I20240503 13:55:41.298942 6537 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 06b84a83a711445f9071c2408ce2ca72 to finish bootstrapping
W20240503 13:55:41.302937 7896 tablet.cc:2369] T 357e0dcea1334d388e4cfcc931fda43a P 06b84a83a711445f9071c2408ce2ca72: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 13:55:41.303599 6537 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 33c9030181f64b7bba94366dbb3ef7d8 to finish bootstrapping
I20240503 13:55:41.307996 6537 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 32cbeb4ffe414abc811ba64fe0c650ad to finish bootstrapping
I20240503 13:55:41.312644 6537 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu with pid 7500
W20240503 13:55:41.397197 8159 tablet.cc:2369] T 357e0dcea1334d388e4cfcc931fda43a P 32cbeb4ffe414abc811ba64fe0c650ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 13:55:41.499636 7960 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:55:41.500130 7696 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:55:41.500648 7828 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:55:41.500707 8092 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:55:41.575306 8092 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:55:41.575881 7828 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:55:41.576782 7696 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:55:41.577046 7960 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:55:41.626588 7762 heartbeater.cc:503] Master 127.6.98.126:44199 was elected leader, sending a full tablet report...
I20240503 13:55:41.694118 8092 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:55:41.694639 7828 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:55:41.695206 7960 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:55:41.695282 7696 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:55:41.822852 8279 consensus_queue.cc:1035] T 357e0dcea1334d388e4cfcc931fda43a P 32cbeb4ffe414abc811ba64fe0c650ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "33c9030181f64b7bba94366dbb3ef7d8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:55:41.828718 8279 consensus_queue.cc:1035] T 357e0dcea1334d388e4cfcc931fda43a P 32cbeb4ffe414abc811ba64fe0c650ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "06b84a83a711445f9071c2408ce2ca72" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:55:41.915354 7828 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:55:41.915649 8092 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:55:41.915910 7960 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:55:41.917232 7696 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:55:41.988833 7960 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:55:41.990568 7828 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:55:41.991614 7696 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:55:41.997336 8092 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:55:42.089116 7828 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:55:42.090436 7960 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:55:42.090703 8092 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:55:42.091390 7696 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:55:42.124064 6537 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu with pid 7633
I20240503 13:55:42.129510 6537 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu with pid 7766
I20240503 13:55:42.134565 6537 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu with pid 7898
I20240503 13:55:42.139304 6537 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu with pid 8029
I20240503 13:55:42.143954 6537 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu with pid 7409
2024-05-03T13:55:42Z chronyd exiting
[ OK ] RebalanceStartCriteriaTest.TabletServerIsDown/0 (2140 ms)
[----------] 1 test from RebalanceStartCriteriaTest (2141 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/0
I20240503 13:55:42.164399 6537 test_util.cc:268] Using random seed: 324072873
I20240503 13:55:42.165316 6537 ts_itest-base.cc:112] Starting cluster with:
I20240503 13:55:42.165359 6537 ts_itest-base.cc:113] --------------
I20240503 13:55:42.165369 6537 ts_itest-base.cc:114] 7 tablet servers
I20240503 13:55:42.165377 6537 ts_itest-base.cc:115] 1 replicas per TS
I20240503 13:55:42.165386 6537 ts_itest-base.cc:116] --------------
2024-05-03T13:55:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T13:55:42Z Disabled control of system clock
I20240503 13:55:42.172662 6537 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
/tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.98.126:36629
--webserver_interface=127.6.98.126
--webserver_port=0
--builtin_ntp_servers=127.6.98.84:33479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.98.126:36629
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240503 13:55:42.275974 8382 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240503 13:55:42.276232 8382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:55:42.276269 8382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:55:42.276329 8382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:55:42.279385 8382 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 13:55:42.279466 8382 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 13:55:42.279506 8382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:55:42.279543 8382 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 13:55:42.279570 8382 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 13:55:42.283218 8382 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.98.84:33479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.6.98.126:36629
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.98.126:36629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.6.98.126
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.8382
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:55:42.284096 8382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:55:42.285245 8382 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:55:42.292313 8387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:55:42.292692 8388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:55:42.292843 8391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:55:42.293399 8382 server_base.cc:1034] running on GCE node
I20240503 13:55:42.293912 8382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:55:42.294824 8382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:55:42.296016 8382 hybrid_clock.cc:648] HybridClock initialized: now 1714744542296000 us; error 33 us; skew 500 ppm
I20240503 13:55:42.297972 8382 webserver.cc:460] Webserver started at http://127.6.98.126:35483/ using document root <none> and password file <none>
I20240503 13:55:42.298518 8382 fs_manager.cc:359] Metadata directory not provided
I20240503 13:55:42.298578 8382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:55:42.298748 8382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:55:42.300416 8382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "9facb48813f1421ca415565225ef9d8b"
format_stamp: "Formatted at 2024-05-03 13:55:42 on dist-test-slave-zg9z"
I20240503 13:55:42.300942 8382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "9facb48813f1421ca415565225ef9d8b"
format_stamp: "Formatted at 2024-05-03 13:55:42 on dist-test-slave-zg9z"
I20240503 13:55:42.304277 8382 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 13:55:42.306599 8397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:55:42.307678 8382 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:55:42.307794 8382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/master-0/wal
uuid: "9facb48813f1421ca415565225ef9d8b"
format_stamp: "Formatted at 2024-05-03 13:55:42 on dist-test-slave-zg9z"
I20240503 13:55:42.307886 8382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:55:42.339289 8382 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:55:42.339987 8382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:55:42.340148 8382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:55:42.346572 8382 rpc_server.cc:307] RPC server started. Bound to: 127.6.98.126:36629
I20240503 13:55:42.346585 8449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.98.126:36629 every 8 connection(s)
I20240503 13:55:42.347561 8382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 13:55:42.347718 6537 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu as pid 8382
I20240503 13:55:42.347832 6537 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 13:55:42.350641 8450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:55:42.355674 8450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9facb48813f1421ca415565225ef9d8b: Bootstrap starting.
I20240503 13:55:42.357805 8450 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9facb48813f1421ca415565225ef9d8b: Neither blocks nor log segments found. Creating new log.
I20240503 13:55:42.358467 8450 log.cc:824] T 00000000000000000000000000000000 P 9facb48813f1421ca415565225ef9d8b: Log is configured to *not* fsync() on all Append() calls
I20240503 13:55:42.360066 8450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9facb48813f1421ca415565225ef9d8b: No bootstrap required, opened a new log
I20240503 13:55:42.362479 8450 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9facb48813f1421ca415565225ef9d8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9facb48813f1421ca415565225ef9d8b" member_type: VOTER last_known_addr { host: "127.6.98.126" port: 36629 } }
I20240503 13:55:42.362663 8450 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9facb48813f1421ca415565225ef9d8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:55:42.362720 8450 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9facb48813f1421ca415565225ef9d8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9facb48813f1421ca415565225ef9d8b, State: Initialized, Role: FOLLOWER
I20240503 13:55:42.363117 8450 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9facb48813f1421ca415565225ef9d8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9facb48813f1421ca415565225ef9d8b" member_type: VOTER last_known_addr { host: "127.6.98.126" port: 36629 } }
I20240503 13:55:42.363240 8450 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9facb48813f1421ca415565225ef9d8b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:55:42.363299 8450 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9facb48813f1421ca415565225ef9d8b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:55:42.363399 8450 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9facb48813f1421ca415565225ef9d8b [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:55:42.364207 8450 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9facb48813f1421ca415565225ef9d8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9facb48813f1421ca415565225ef9d8b" member_type: VOTER last_known_addr { host: "127.6.98.126" port: 36629 } }
I20240503 13:55:42.364528 8450 leader_election.cc:302] T 00000000000000000000000000000000 P 9facb48813f1421ca415565225ef9d8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9facb48813f1421ca415565225ef9d8b; no voters:
I20240503 13:55:42.364821 8450 leader_election.cc:288] T 00000000000000000000000000000000 P 9facb48813f1421ca415565225ef9d8b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 13:55:42.365077 8455 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9facb48813f1421ca415565225ef9d8b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:55:42.365267 8455 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9facb48813f1421ca415565225ef9d8b [term 1 LEADER]: Becoming Leader. State: Replica: 9facb48813f1421ca415565225ef9d8b, State: Running, Role: LEADER
I20240503 13:55:42.365556 8455 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9facb48813f1421ca415565225ef9d8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9facb48813f1421ca415565225ef9d8b" member_type: VOTER last_known_addr { host: "127.6.98.126" port: 36629 } }
I20240503 13:55:42.365842 8450 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9facb48813f1421ca415565225ef9d8b [sys.catalog]: configured and running, proceeding with master startup.
I20240503 13:55:42.367014 8456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9facb48813f1421ca415565225ef9d8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9facb48813f1421ca415565225ef9d8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9facb48813f1421ca415565225ef9d8b" member_type: VOTER last_known_addr { host: "127.6.98.126" port: 36629 } } }
I20240503 13:55:42.367142 8456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9facb48813f1421ca415565225ef9d8b [sys.catalog]: This master's current role is: LEADER
I20240503 13:55:42.367005 8457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9facb48813f1421ca415565225ef9d8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9facb48813f1421ca415565225ef9d8b. Latest consensus state: current_term: 1 leader_uuid: "9facb48813f1421ca415565225ef9d8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9facb48813f1421ca415565225ef9d8b" member_type: VOTER last_known_addr { host: "127.6.98.126" port: 36629 } } }
I20240503 13:55:42.367290 8457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9facb48813f1421ca415565225ef9d8b [sys.catalog]: This master's current role is: LEADER
I20240503 13:55:42.367501 8465 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 13:55:42.370141 8465 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 13:55:42.374886 8465 catalog_manager.cc:1347] Generated new cluster ID: 01fada3df0d04cb7bc02871d848e8e6a
I20240503 13:55:42.374961 8465 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 13:55:42.380160 8465 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 13:55:42.381045 8465 catalog_manager.cc:1504] Loading token signing keys...
I20240503 13:55:42.392302 8465 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 9facb48813f1421ca415565225ef9d8b: Generated new TSK 0
I20240503 13:55:42.393011 8465 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 13:55:42.402575 6537 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
/tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.98.65:0
--local_ip_for_outbound_sockets=127.6.98.65
--webserver_interface=127.6.98.65
--webserver_port=0
--tserver_master_addrs=127.6.98.126:36629
--builtin_ntp_servers=127.6.98.84:33479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 13:55:42.512677 8474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:55:42.512897 8474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:55:42.512943 8474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:55:42.515985 8474 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 13:55:42.516053 8474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:55:42.516131 8474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.98.65
I20240503 13:55:42.519819 8474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.98.84:33479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.98.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.6.98.65
--webserver_port=0
--tserver_master_addrs=127.6.98.126:36629
--never_fsync=true
--heap_profile_path=/tmp/kudu.8474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.98.65
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:55:42.520769 8474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:55:42.521883 8474 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:55:42.529932 8482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:55:42.530180 8479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:55:42.530339 8480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:55:42.530753 8474 server_base.cc:1034] running on GCE node
I20240503 13:55:42.531173 8474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:55:42.531769 8474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:55:42.532943 8474 hybrid_clock.cc:648] HybridClock initialized: now 1714744542532916 us; error 36 us; skew 500 ppm
I20240503 13:55:42.534828 8474 webserver.cc:460] Webserver started at http://127.6.98.65:43983/ using document root <none> and password file <none>
I20240503 13:55:42.535399 8474 fs_manager.cc:359] Metadata directory not provided
I20240503 13:55:42.535454 8474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:55:42.535624 8474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:55:42.537315 8474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "a7346c212aa54138970650af7f721d0e"
format_stamp: "Formatted at 2024-05-03 13:55:42 on dist-test-slave-zg9z"
I20240503 13:55:42.537773 8474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "a7346c212aa54138970650af7f721d0e"
format_stamp: "Formatted at 2024-05-03 13:55:42 on dist-test-slave-zg9z"
I20240503 13:55:42.541035 8474 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 13:55:42.543151 8488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:55:42.544176 8474 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:55:42.544366 8474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "a7346c212aa54138970650af7f721d0e"
format_stamp: "Formatted at 2024-05-03 13:55:42 on dist-test-slave-zg9z"
I20240503 13:55:42.544473 8474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:55:42.562762 8474 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:55:42.563439 8474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:55:42.563593 8474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:55:42.564136 8474 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:55:42.565104 8474 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:55:42.565164 8474 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:55:42.565223 8474 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:55:42.565253 8474 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:55:42.573446 8474 rpc_server.cc:307] RPC server started. Bound to: 127.6.98.65:37097
I20240503 13:55:42.573480 8601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.98.65:37097 every 8 connection(s)
I20240503 13:55:42.574347 8474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 13:55:42.578712 6537 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu as pid 8474
I20240503 13:55:42.578837 6537 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 13:55:42.580863 6537 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
/tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.98.66:0
--local_ip_for_outbound_sockets=127.6.98.66
--webserver_interface=127.6.98.66
--webserver_port=0
--tserver_master_addrs=127.6.98.126:36629
--builtin_ntp_servers=127.6.98.84:33479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 13:55:42.583065 8602 heartbeater.cc:347] Connected to a master server at 127.6.98.126:36629
I20240503 13:55:42.583278 8602 heartbeater.cc:464] Registering TS with master...
I20240503 13:55:42.583706 8602 heartbeater.cc:511] Master 127.6.98.126:36629 requested a full tablet report, sending...
I20240503 13:55:42.585048 8414 ts_manager.cc:194] Registered new tserver with Master: a7346c212aa54138970650af7f721d0e (127.6.98.65:37097)
I20240503 13:55:42.586166 8414 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.98.65:36069
W20240503 13:55:42.687912 8605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:55:42.688138 8605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:55:42.688182 8605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:55:42.691365 8605 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 13:55:42.691429 8605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:55:42.691511 8605 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.98.66
I20240503 13:55:42.695228 8605 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.98.84:33479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.98.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.6.98.66
--webserver_port=0
--tserver_master_addrs=127.6.98.126:36629
--never_fsync=true
--heap_profile_path=/tmp/kudu.8605
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.98.66
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:55:42.696143 8605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:55:42.697275 8605 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:55:42.705272 8613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:55:42.705806 8610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:55:42.706161 8611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:55:42.706610 8605 server_base.cc:1034] running on GCE node
I20240503 13:55:42.707005 8605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:55:42.707594 8605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:55:42.708745 8605 hybrid_clock.cc:648] HybridClock initialized: now 1714744542708730 us; error 32 us; skew 500 ppm
I20240503 13:55:42.710678 8605 webserver.cc:460] Webserver started at http://127.6.98.66:42433/ using document root <none> and password file <none>
I20240503 13:55:42.711263 8605 fs_manager.cc:359] Metadata directory not provided
I20240503 13:55:42.711318 8605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:55:42.711501 8605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:55:42.713209 8605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "30acad1886794455a2b5617ba6da9aa1"
format_stamp: "Formatted at 2024-05-03 13:55:42 on dist-test-slave-zg9z"
I20240503 13:55:42.713680 8605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "30acad1886794455a2b5617ba6da9aa1"
format_stamp: "Formatted at 2024-05-03 13:55:42 on dist-test-slave-zg9z"
I20240503 13:55:42.717080 8605 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 13:55:42.719301 8619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:55:42.720396 8605 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 13:55:42.720537 8605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "30acad1886794455a2b5617ba6da9aa1"
format_stamp: "Formatted at 2024-05-03 13:55:42 on dist-test-slave-zg9z"
I20240503 13:55:42.720634 8605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:55:42.742702 8605 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:55:42.743398 8605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:55:42.743559 8605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:55:42.744115 8605 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:55:42.745112 8605 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:55:42.745162 8605 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:55:42.745216 8605 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:55:42.745242 8605 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:55:42.753856 8605 rpc_server.cc:307] RPC server started. Bound to: 127.6.98.66:39533
I20240503 13:55:42.753906 8732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.98.66:39533 every 8 connection(s)
I20240503 13:55:42.754838 8605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 13:55:42.756675 6537 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu as pid 8605
I20240503 13:55:42.756781 6537 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 13:55:42.759119 6537 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
/tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.98.67:0
--local_ip_for_outbound_sockets=127.6.98.67
--webserver_interface=127.6.98.67
--webserver_port=0
--tserver_master_addrs=127.6.98.126:36629
--builtin_ntp_servers=127.6.98.84:33479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 13:55:42.765105 8733 heartbeater.cc:347] Connected to a master server at 127.6.98.126:36629
I20240503 13:55:42.765395 8733 heartbeater.cc:464] Registering TS with master...
I20240503 13:55:42.766062 8733 heartbeater.cc:511] Master 127.6.98.126:36629 requested a full tablet report, sending...
I20240503 13:55:42.767200 8414 ts_manager.cc:194] Registered new tserver with Master: 30acad1886794455a2b5617ba6da9aa1 (127.6.98.66:39533)
I20240503 13:55:42.768020 8414 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.98.66:60893
W20240503 13:55:42.871874 8736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:55:42.872098 8736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:55:42.872141 8736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:55:42.875236 8736 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 13:55:42.875308 8736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:55:42.875384 8736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.98.67
I20240503 13:55:42.879096 8736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.98.84:33479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.98.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.6.98.67
--webserver_port=0
--tserver_master_addrs=127.6.98.126:36629
--never_fsync=true
--heap_profile_path=/tmp/kudu.8736
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.98.67
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:55:42.880008 8736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:55:42.881165 8736 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:55:42.888967 8744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:55:42.889292 8741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:55:42.889791 8742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:55:42.890383 8736 server_base.cc:1034] running on GCE node
I20240503 13:55:42.890785 8736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:55:42.891397 8736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:55:42.892573 8736 hybrid_clock.cc:648] HybridClock initialized: now 1714744542892546 us; error 35 us; skew 500 ppm
I20240503 13:55:42.894774 8736 webserver.cc:460] Webserver started at http://127.6.98.67:34021/ using document root <none> and password file <none>
I20240503 13:55:42.895349 8736 fs_manager.cc:359] Metadata directory not provided
I20240503 13:55:42.895406 8736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:55:42.895589 8736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:55:42.897379 8736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "45bdf7ba415141cd85008dca119d1cf9"
format_stamp: "Formatted at 2024-05-03 13:55:42 on dist-test-slave-zg9z"
I20240503 13:55:42.899783 8736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "45bdf7ba415141cd85008dca119d1cf9"
format_stamp: "Formatted at 2024-05-03 13:55:42 on dist-test-slave-zg9z"
I20240503 13:55:42.903575 8736 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20240503 13:55:42.906014 8750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:55:42.907095 8736 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 13:55:42.907223 8736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "45bdf7ba415141cd85008dca119d1cf9"
format_stamp: "Formatted at 2024-05-03 13:55:42 on dist-test-slave-zg9z"
I20240503 13:55:42.907312 8736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:55:42.929540 8736 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:55:42.930228 8736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:55:42.930380 8736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:55:42.930927 8736 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:55:42.931828 8736 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:55:42.931874 8736 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:55:42.931934 8736 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:55:42.931958 8736 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:55:42.940222 8736 rpc_server.cc:307] RPC server started. Bound to: 127.6.98.67:40171
I20240503 13:55:42.940236 8863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.98.67:40171 every 8 connection(s)
I20240503 13:55:42.941172 8736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 13:55:42.945773 6537 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu as pid 8736
I20240503 13:55:42.945914 6537 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 13:55:42.948086 6537 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
/tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.98.68:0
--local_ip_for_outbound_sockets=127.6.98.68
--webserver_interface=127.6.98.68
--webserver_port=0
--tserver_master_addrs=127.6.98.126:36629
--builtin_ntp_servers=127.6.98.84:33479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 13:55:42.950237 8864 heartbeater.cc:347] Connected to a master server at 127.6.98.126:36629
I20240503 13:55:42.950479 8864 heartbeater.cc:464] Registering TS with master...
I20240503 13:55:42.950917 8864 heartbeater.cc:511] Master 127.6.98.126:36629 requested a full tablet report, sending...
I20240503 13:55:42.951885 8414 ts_manager.cc:194] Registered new tserver with Master: 45bdf7ba415141cd85008dca119d1cf9 (127.6.98.67:40171)
I20240503 13:55:42.952667 8414 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.98.67:52267
W20240503 13:55:43.063938 8867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:55:43.064170 8867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:55:43.064213 8867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:55:43.067354 8867 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 13:55:43.067423 8867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:55:43.067490 8867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.98.68
I20240503 13:55:43.071111 8867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.98.84:33479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.98.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.6.98.68
--webserver_port=0
--tserver_master_addrs=127.6.98.126:36629
--never_fsync=true
--heap_profile_path=/tmp/kudu.8867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.98.68
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:55:43.072005 8867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:55:43.073117 8867 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:55:43.081233 8872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:55:43.081609 8873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:55:43.081750 8875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:55:43.082269 8867 server_base.cc:1034] running on GCE node
I20240503 13:55:43.082643 8867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:55:43.083176 8867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:55:43.084323 8867 hybrid_clock.cc:648] HybridClock initialized: now 1714744543084310 us; error 27 us; skew 500 ppm
I20240503 13:55:43.086143 8867 webserver.cc:460] Webserver started at http://127.6.98.68:45725/ using document root <none> and password file <none>
I20240503 13:55:43.086704 8867 fs_manager.cc:359] Metadata directory not provided
I20240503 13:55:43.086760 8867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:55:43.086926 8867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:55:43.088579 8867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "91d1a791a7504f71a4785f6aefe26f1b"
format_stamp: "Formatted at 2024-05-03 13:55:43 on dist-test-slave-zg9z"
I20240503 13:55:43.089022 8867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "91d1a791a7504f71a4785f6aefe26f1b"
format_stamp: "Formatted at 2024-05-03 13:55:43 on dist-test-slave-zg9z"
I20240503 13:55:43.092175 8867 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 13:55:43.094363 8881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:55:43.095338 8867 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:55:43.095438 8867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "91d1a791a7504f71a4785f6aefe26f1b"
format_stamp: "Formatted at 2024-05-03 13:55:43 on dist-test-slave-zg9z"
I20240503 13:55:43.095539 8867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:55:43.124022 8867 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:55:43.124805 8867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:55:43.124975 8867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:55:43.125535 8867 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:55:43.126418 8867 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:55:43.126464 8867 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:55:43.126516 8867 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:55:43.126542 8867 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:55:43.134693 8867 rpc_server.cc:307] RPC server started. Bound to: 127.6.98.68:40653
I20240503 13:55:43.134723 8994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.98.68:40653 every 8 connection(s)
I20240503 13:55:43.135581 8867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 13:55:43.143687 8995 heartbeater.cc:347] Connected to a master server at 127.6.98.126:36629
I20240503 13:55:43.143985 8995 heartbeater.cc:464] Registering TS with master...
I20240503 13:55:43.144421 6537 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu as pid 8867
I20240503 13:55:43.144579 6537 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 13:55:43.144603 8995 heartbeater.cc:511] Master 127.6.98.126:36629 requested a full tablet report, sending...
I20240503 13:55:43.145654 8414 ts_manager.cc:194] Registered new tserver with Master: 91d1a791a7504f71a4785f6aefe26f1b (127.6.98.68:40653)
I20240503 13:55:43.146345 8414 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.98.68:38975
I20240503 13:55:43.146929 6537 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
/tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.98.69:0
--local_ip_for_outbound_sockets=127.6.98.69
--webserver_interface=127.6.98.69
--webserver_port=0
--tserver_master_addrs=127.6.98.126:36629
--builtin_ntp_servers=127.6.98.84:33479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 13:55:43.255137 8998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:55:43.255358 8998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:55:43.255398 8998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:55:43.258461 8998 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 13:55:43.258539 8998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:55:43.258606 8998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.98.69
I20240503 13:55:43.262240 8998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.98.84:33479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.98.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.6.98.69
--webserver_port=0
--tserver_master_addrs=127.6.98.126:36629
--never_fsync=true
--heap_profile_path=/tmp/kudu.8998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.98.69
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:55:43.263095 8998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:55:43.264171 8998 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:55:43.272333 9006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:55:43.272333 9003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:55:43.272756 9004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:55:43.273278 8998 server_base.cc:1034] running on GCE node
I20240503 13:55:43.273635 8998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:55:43.274199 8998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:55:43.275350 8998 hybrid_clock.cc:648] HybridClock initialized: now 1714744543275331 us; error 39 us; skew 500 ppm
I20240503 13:55:43.277222 8998 webserver.cc:460] Webserver started at http://127.6.98.69:41739/ using document root <none> and password file <none>
I20240503 13:55:43.277753 8998 fs_manager.cc:359] Metadata directory not provided
I20240503 13:55:43.277808 8998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:55:43.277989 8998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:55:43.279599 8998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "6a0591a0f0b64c418af287c96b5b47b8"
format_stamp: "Formatted at 2024-05-03 13:55:43 on dist-test-slave-zg9z"
I20240503 13:55:43.280067 8998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "6a0591a0f0b64c418af287c96b5b47b8"
format_stamp: "Formatted at 2024-05-03 13:55:43 on dist-test-slave-zg9z"
I20240503 13:55:43.283337 8998 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 13:55:43.285609 9012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:55:43.286600 8998 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 13:55:43.286703 8998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "6a0591a0f0b64c418af287c96b5b47b8"
format_stamp: "Formatted at 2024-05-03 13:55:43 on dist-test-slave-zg9z"
I20240503 13:55:43.286790 8998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:55:43.297830 8998 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:55:43.298453 8998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:55:43.298611 8998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:55:43.299151 8998 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:55:43.300063 8998 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:55:43.300112 8998 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:55:43.300161 8998 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:55:43.300187 8998 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:55:43.308424 8998 rpc_server.cc:307] RPC server started. Bound to: 127.6.98.69:33575
I20240503 13:55:43.308533 9125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.98.69:33575 every 8 connection(s)
I20240503 13:55:43.309412 8998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 13:55:43.312741 6537 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu as pid 8998
I20240503 13:55:43.312863 6537 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 13:55:43.315230 6537 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
/tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.98.70:0
--local_ip_for_outbound_sockets=127.6.98.70
--webserver_interface=127.6.98.70
--webserver_port=0
--tserver_master_addrs=127.6.98.126:36629
--builtin_ntp_servers=127.6.98.84:33479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 13:55:43.319146 9126 heartbeater.cc:347] Connected to a master server at 127.6.98.126:36629
I20240503 13:55:43.319464 9126 heartbeater.cc:464] Registering TS with master...
I20240503 13:55:43.319983 9126 heartbeater.cc:511] Master 127.6.98.126:36629 requested a full tablet report, sending...
I20240503 13:55:43.320971 8414 ts_manager.cc:194] Registered new tserver with Master: 6a0591a0f0b64c418af287c96b5b47b8 (127.6.98.69:33575)
I20240503 13:55:43.321601 8414 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.98.69:40993
W20240503 13:55:43.424049 9129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:55:43.424304 9129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:55:43.424355 9129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:55:43.427578 9129 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 13:55:43.427644 9129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:55:43.427721 9129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.98.70
I20240503 13:55:43.431491 9129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.98.84:33479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.98.70:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.6.98.70
--webserver_port=0
--tserver_master_addrs=127.6.98.126:36629
--never_fsync=true
--heap_profile_path=/tmp/kudu.9129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.98.70
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:55:43.432425 9129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:55:43.433619 9129 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:55:43.441956 9134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:55:43.442319 9137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:55:43.442445 9135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:55:43.442983 9129 server_base.cc:1034] running on GCE node
I20240503 13:55:43.443372 9129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:55:43.443966 9129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:55:43.445119 9129 hybrid_clock.cc:648] HybridClock initialized: now 1714744543445092 us; error 41 us; skew 500 ppm
I20240503 13:55:43.447082 9129 webserver.cc:460] Webserver started at http://127.6.98.70:42253/ using document root <none> and password file <none>
I20240503 13:55:43.447674 9129 fs_manager.cc:359] Metadata directory not provided
I20240503 13:55:43.447739 9129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:55:43.447940 9129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:55:43.449765 9129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "a075f746a2e04544931d75c3dc463ef3"
format_stamp: "Formatted at 2024-05-03 13:55:43 on dist-test-slave-zg9z"
I20240503 13:55:43.450237 9129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "a075f746a2e04544931d75c3dc463ef3"
format_stamp: "Formatted at 2024-05-03 13:55:43 on dist-test-slave-zg9z"
I20240503 13:55:43.453596 9129 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 13:55:43.456048 9143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:55:43.457144 9129 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 13:55:43.457259 9129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "a075f746a2e04544931d75c3dc463ef3"
format_stamp: "Formatted at 2024-05-03 13:55:43 on dist-test-slave-zg9z"
I20240503 13:55:43.457381 9129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:55:43.475282 9129 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:55:43.475953 9129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:55:43.476104 9129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:55:43.476682 9129 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:55:43.477602 9129 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:55:43.477645 9129 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:55:43.477699 9129 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:55:43.477722 9129 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:55:43.485918 9129 rpc_server.cc:307] RPC server started. Bound to: 127.6.98.70:33391
I20240503 13:55:43.485944 9256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.98.70:33391 every 8 connection(s)
I20240503 13:55:43.486816 9129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 13:55:43.491185 6537 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu as pid 9129
I20240503 13:55:43.491305 6537 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 13:55:43.493484 6537 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
/tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.98.71:0
--local_ip_for_outbound_sockets=127.6.98.71
--webserver_interface=127.6.98.71
--webserver_port=0
--tserver_master_addrs=127.6.98.126:36629
--builtin_ntp_servers=127.6.98.84:33479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 13:55:43.495954 9257 heartbeater.cc:347] Connected to a master server at 127.6.98.126:36629
I20240503 13:55:43.496217 9257 heartbeater.cc:464] Registering TS with master...
I20240503 13:55:43.496803 9257 heartbeater.cc:511] Master 127.6.98.126:36629 requested a full tablet report, sending...
I20240503 13:55:43.497944 8414 ts_manager.cc:194] Registered new tserver with Master: a075f746a2e04544931d75c3dc463ef3 (127.6.98.70:33391)
I20240503 13:55:43.498510 8414 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.98.70:50623
I20240503 13:55:43.588269 8602 heartbeater.cc:503] Master 127.6.98.126:36629 was elected leader, sending a full tablet report...
W20240503 13:55:43.603235 9260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:55:43.603480 9260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:55:43.603542 9260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:55:43.606745 9260 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 13:55:43.606834 9260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:55:43.606931 9260 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.98.71
I20240503 13:55:43.610716 9260 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.98.84:33479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.98.71:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.6.98.71
--webserver_port=0
--tserver_master_addrs=127.6.98.126:36629
--never_fsync=true
--heap_profile_path=/tmp/kudu.9260
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.98.71
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:55:43.611726 9260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:55:43.612995 9260 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:55:43.622956 9268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:55:43.622956 9266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:55:43.623289 9265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:55:43.623929 9260 server_base.cc:1034] running on GCE node
I20240503 13:55:43.624454 9260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:55:43.625324 9260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:55:43.626540 9260 hybrid_clock.cc:648] HybridClock initialized: now 1714744543626515 us; error 45 us; skew 500 ppm
I20240503 13:55:43.637135 9260 webserver.cc:460] Webserver started at http://127.6.98.71:39801/ using document root <none> and password file <none>
I20240503 13:55:43.637750 9260 fs_manager.cc:359] Metadata directory not provided
I20240503 13:55:43.637814 9260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:55:43.638010 9260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:55:43.639719 9260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "44f093b99a6a403f8e092124ba9fea4c"
format_stamp: "Formatted at 2024-05-03 13:55:43 on dist-test-slave-zg9z"
I20240503 13:55:43.640204 9260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "44f093b99a6a403f8e092124ba9fea4c"
format_stamp: "Formatted at 2024-05-03 13:55:43 on dist-test-slave-zg9z"
I20240503 13:55:43.643627 9260 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20240503 13:55:43.646087 9274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:55:43.647228 9260 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 13:55:43.647342 9260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "44f093b99a6a403f8e092124ba9fea4c"
format_stamp: "Formatted at 2024-05-03 13:55:43 on dist-test-slave-zg9z"
I20240503 13:55:43.647445 9260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:55:43.671191 9260 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:55:43.671850 9260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:55:43.672012 9260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:55:43.672574 9260 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:55:43.673466 9260 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:55:43.673511 9260 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:55:43.673563 9260 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:55:43.673590 9260 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:55:43.681875 9260 rpc_server.cc:307] RPC server started. Bound to: 127.6.98.71:39765
I20240503 13:55:43.681905 9387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.98.71:39765 every 8 connection(s)
I20240503 13:55:43.682788 9260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 13:55:43.690198 6537 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu as pid 9260
I20240503 13:55:43.690313 6537 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 13:55:43.691709 9388 heartbeater.cc:347] Connected to a master server at 127.6.98.126:36629
I20240503 13:55:43.692014 9388 heartbeater.cc:464] Registering TS with master...
I20240503 13:55:43.692629 9388 heartbeater.cc:511] Master 127.6.98.126:36629 requested a full tablet report, sending...
I20240503 13:55:43.693573 8409 ts_manager.cc:194] Registered new tserver with Master: 44f093b99a6a403f8e092124ba9fea4c (127.6.98.71:39765)
I20240503 13:55:43.694118 8409 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.98.71:43129
I20240503 13:55:43.704887 6537 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240503 13:55:43.714064 8409 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45454:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 13:55:43.730341 8929 tablet_service.cc:1467] Processing CreateTablet for tablet 331586b138a94590b0a06e6fec84bf4a (DEFAULT_TABLE table=TestTable [id=8b4072d9c1124181a340fd62ceeea722]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 13:55:43.731215 8929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 331586b138a94590b0a06e6fec84bf4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:55:43.736167 9402 tablet_bootstrap.cc:492] T 331586b138a94590b0a06e6fec84bf4a P 91d1a791a7504f71a4785f6aefe26f1b: Bootstrap starting.
I20240503 13:55:43.738320 9402 tablet_bootstrap.cc:654] T 331586b138a94590b0a06e6fec84bf4a P 91d1a791a7504f71a4785f6aefe26f1b: Neither blocks nor log segments found. Creating new log.
I20240503 13:55:43.739003 9402 log.cc:824] T 331586b138a94590b0a06e6fec84bf4a P 91d1a791a7504f71a4785f6aefe26f1b: Log is configured to *not* fsync() on all Append() calls
I20240503 13:55:43.740655 9402 tablet_bootstrap.cc:492] T 331586b138a94590b0a06e6fec84bf4a P 91d1a791a7504f71a4785f6aefe26f1b: No bootstrap required, opened a new log
I20240503 13:55:43.740833 9402 ts_tablet_manager.cc:1367] T 331586b138a94590b0a06e6fec84bf4a P 91d1a791a7504f71a4785f6aefe26f1b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 13:55:43.743306 9402 raft_consensus.cc:348] T 331586b138a94590b0a06e6fec84bf4a P 91d1a791a7504f71a4785f6aefe26f1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91d1a791a7504f71a4785f6aefe26f1b" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 40653 } }
I20240503 13:55:43.743505 9402 raft_consensus.cc:374] T 331586b138a94590b0a06e6fec84bf4a P 91d1a791a7504f71a4785f6aefe26f1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:55:43.743541 9402 raft_consensus.cc:729] T 331586b138a94590b0a06e6fec84bf4a P 91d1a791a7504f71a4785f6aefe26f1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d1a791a7504f71a4785f6aefe26f1b, State: Initialized, Role: FOLLOWER
I20240503 13:55:43.743971 9402 consensus_queue.cc:260] T 331586b138a94590b0a06e6fec84bf4a P 91d1a791a7504f71a4785f6aefe26f1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91d1a791a7504f71a4785f6aefe26f1b" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 40653 } }
I20240503 13:55:43.744096 9402 raft_consensus.cc:388] T 331586b138a94590b0a06e6fec84bf4a P 91d1a791a7504f71a4785f6aefe26f1b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:55:43.744163 9402 raft_consensus.cc:482] T 331586b138a94590b0a06e6fec84bf4a P 91d1a791a7504f71a4785f6aefe26f1b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:55:43.744267 9402 raft_consensus.cc:3037] T 331586b138a94590b0a06e6fec84bf4a P 91d1a791a7504f71a4785f6aefe26f1b [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:55:43.745528 9402 raft_consensus.cc:504] T 331586b138a94590b0a06e6fec84bf4a P 91d1a791a7504f71a4785f6aefe26f1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91d1a791a7504f71a4785f6aefe26f1b" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 40653 } }
I20240503 13:55:43.745860 9402 leader_election.cc:302] T 331586b138a94590b0a06e6fec84bf4a P 91d1a791a7504f71a4785f6aefe26f1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 91d1a791a7504f71a4785f6aefe26f1b; no voters:
I20240503 13:55:43.746140 9402 leader_election.cc:288] T 331586b138a94590b0a06e6fec84bf4a P 91d1a791a7504f71a4785f6aefe26f1b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 13:55:43.746273 9404 raft_consensus.cc:2781] T 331586b138a94590b0a06e6fec84bf4a P 91d1a791a7504f71a4785f6aefe26f1b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:55:43.746481 9404 raft_consensus.cc:686] T 331586b138a94590b0a06e6fec84bf4a P 91d1a791a7504f71a4785f6aefe26f1b [term 1 LEADER]: Becoming Leader. State: Replica: 91d1a791a7504f71a4785f6aefe26f1b, State: Running, Role: LEADER
I20240503 13:55:43.746802 9404 consensus_queue.cc:237] T 331586b138a94590b0a06e6fec84bf4a P 91d1a791a7504f71a4785f6aefe26f1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91d1a791a7504f71a4785f6aefe26f1b" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 40653 } }
I20240503 13:55:43.746941 8995 heartbeater.cc:503] Master 127.6.98.126:36629 was elected leader, sending a full tablet report...
I20240503 13:55:43.747018 9402 ts_tablet_manager.cc:1398] T 331586b138a94590b0a06e6fec84bf4a P 91d1a791a7504f71a4785f6aefe26f1b: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240503 13:55:43.749979 8409 catalog_manager.cc:5484] T 331586b138a94590b0a06e6fec84bf4a P 91d1a791a7504f71a4785f6aefe26f1b reported cstate change: term changed from 0 to 1, leader changed from <none> to 91d1a791a7504f71a4785f6aefe26f1b (127.6.98.68). New cstate: current_term: 1 leader_uuid: "91d1a791a7504f71a4785f6aefe26f1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91d1a791a7504f71a4785f6aefe26f1b" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 40653 } } }
I20240503 13:55:43.757467 6537 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240503 13:55:43.758913 6537 ts_itest-base.cc:233] Waiting for 0 tablets on tserver a7346c212aa54138970650af7f721d0e to finish bootstrapping
I20240503 13:55:43.765044 6537 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 30acad1886794455a2b5617ba6da9aa1 to finish bootstrapping
I20240503 13:55:43.770483 8733 heartbeater.cc:503] Master 127.6.98.126:36629 was elected leader, sending a full tablet report...
I20240503 13:55:43.770928 6537 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 45bdf7ba415141cd85008dca119d1cf9 to finish bootstrapping
I20240503 13:55:43.776397 6537 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 91d1a791a7504f71a4785f6aefe26f1b to finish bootstrapping
I20240503 13:55:43.781164 6537 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 6a0591a0f0b64c418af287c96b5b47b8 to finish bootstrapping
I20240503 13:55:43.786855 6537 ts_itest-base.cc:233] Waiting for 0 tablets on tserver a075f746a2e04544931d75c3dc463ef3 to finish bootstrapping
I20240503 13:55:43.792532 6537 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 44f093b99a6a403f8e092124ba9fea4c to finish bootstrapping
I20240503 13:55:43.798239 6537 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu with pid 8736
I20240503 13:55:43.803246 6537 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu with pid 8867
I20240503 13:55:43.808015 6537 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu with pid 8998
I20240503 13:55:43.812698 6537 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu with pid 9129
I20240503 13:55:43.817003 6537 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu with pid 9260
I20240503 13:55:47.572899 8409 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45454:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:55:47.585995 8536 tablet_service.cc:1467] Processing CreateTablet for tablet a9cea7e1421c43fb85282e212e39715d (DEFAULT_TABLE table=rebalance_test_table_0 [id=477d6119baf54f00be410eea6e66b45d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:55:47.586000 8667 tablet_service.cc:1467] Processing CreateTablet for tablet 74d4351322b4423d81178462b8286342 (DEFAULT_TABLE table=rebalance_test_table_0 [id=477d6119baf54f00be410eea6e66b45d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:55:47.586000 8535 tablet_service.cc:1467] Processing CreateTablet for tablet 30ab7f35240e47ffb4c95fc56dc48fd3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=477d6119baf54f00be410eea6e66b45d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:55:47.587008 8667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74d4351322b4423d81178462b8286342. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:55:47.587004 8535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30ab7f35240e47ffb4c95fc56dc48fd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:55:47.587368 8536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9cea7e1421c43fb85282e212e39715d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:55:47.592080 9424 tablet_bootstrap.cc:492] T a9cea7e1421c43fb85282e212e39715d P a7346c212aa54138970650af7f721d0e: Bootstrap starting.
I20240503 13:55:47.592381 9425 tablet_bootstrap.cc:492] T 74d4351322b4423d81178462b8286342 P 30acad1886794455a2b5617ba6da9aa1: Bootstrap starting.
I20240503 13:55:47.594074 9424 tablet_bootstrap.cc:654] T a9cea7e1421c43fb85282e212e39715d P a7346c212aa54138970650af7f721d0e: Neither blocks nor log segments found. Creating new log.
I20240503 13:55:47.594074 9425 tablet_bootstrap.cc:654] T 74d4351322b4423d81178462b8286342 P 30acad1886794455a2b5617ba6da9aa1: Neither blocks nor log segments found. Creating new log.
I20240503 13:55:47.594694 9425 log.cc:824] T 74d4351322b4423d81178462b8286342 P 30acad1886794455a2b5617ba6da9aa1: Log is configured to *not* fsync() on all Append() calls
I20240503 13:55:47.594694 9424 log.cc:824] T a9cea7e1421c43fb85282e212e39715d P a7346c212aa54138970650af7f721d0e: Log is configured to *not* fsync() on all Append() calls
I20240503 13:55:47.596372 9424 tablet_bootstrap.cc:492] T a9cea7e1421c43fb85282e212e39715d P a7346c212aa54138970650af7f721d0e: No bootstrap required, opened a new log
I20240503 13:55:47.596576 9424 ts_tablet_manager.cc:1367] T a9cea7e1421c43fb85282e212e39715d P a7346c212aa54138970650af7f721d0e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 13:55:47.596912 9425 tablet_bootstrap.cc:492] T 74d4351322b4423d81178462b8286342 P 30acad1886794455a2b5617ba6da9aa1: No bootstrap required, opened a new log
I20240503 13:55:47.597075 9425 ts_tablet_manager.cc:1367] T 74d4351322b4423d81178462b8286342 P 30acad1886794455a2b5617ba6da9aa1: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 13:55:47.599409 9424 raft_consensus.cc:348] T a9cea7e1421c43fb85282e212e39715d P a7346c212aa54138970650af7f721d0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } }
I20240503 13:55:47.599584 9424 raft_consensus.cc:374] T a9cea7e1421c43fb85282e212e39715d P a7346c212aa54138970650af7f721d0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:55:47.599619 9424 raft_consensus.cc:729] T a9cea7e1421c43fb85282e212e39715d P a7346c212aa54138970650af7f721d0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7346c212aa54138970650af7f721d0e, State: Initialized, Role: FOLLOWER
I20240503 13:55:47.599903 9425 raft_consensus.cc:348] T 74d4351322b4423d81178462b8286342 P 30acad1886794455a2b5617ba6da9aa1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } }
I20240503 13:55:47.600013 9424 consensus_queue.cc:260] T a9cea7e1421c43fb85282e212e39715d P a7346c212aa54138970650af7f721d0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } }
I20240503 13:55:47.600080 9425 raft_consensus.cc:374] T 74d4351322b4423d81178462b8286342 P 30acad1886794455a2b5617ba6da9aa1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:55:47.600128 9425 raft_consensus.cc:729] T 74d4351322b4423d81178462b8286342 P 30acad1886794455a2b5617ba6da9aa1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30acad1886794455a2b5617ba6da9aa1, State: Initialized, Role: FOLLOWER
I20240503 13:55:47.600154 9424 raft_consensus.cc:388] T a9cea7e1421c43fb85282e212e39715d P a7346c212aa54138970650af7f721d0e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:55:47.600215 9424 raft_consensus.cc:482] T a9cea7e1421c43fb85282e212e39715d P a7346c212aa54138970650af7f721d0e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:55:47.600322 9424 raft_consensus.cc:3037] T a9cea7e1421c43fb85282e212e39715d P a7346c212aa54138970650af7f721d0e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:55:47.600550 9425 consensus_queue.cc:260] T 74d4351322b4423d81178462b8286342 P 30acad1886794455a2b5617ba6da9aa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } }
I20240503 13:55:47.600674 9425 raft_consensus.cc:388] T 74d4351322b4423d81178462b8286342 P 30acad1886794455a2b5617ba6da9aa1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:55:47.600732 9425 raft_consensus.cc:482] T 74d4351322b4423d81178462b8286342 P 30acad1886794455a2b5617ba6da9aa1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:55:47.600821 9425 raft_consensus.cc:3037] T 74d4351322b4423d81178462b8286342 P 30acad1886794455a2b5617ba6da9aa1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:55:47.601178 9424 raft_consensus.cc:504] T a9cea7e1421c43fb85282e212e39715d P a7346c212aa54138970650af7f721d0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } }
I20240503 13:55:47.601475 9424 leader_election.cc:302] T a9cea7e1421c43fb85282e212e39715d P a7346c212aa54138970650af7f721d0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7346c212aa54138970650af7f721d0e; no voters:
I20240503 13:55:47.601547 9425 raft_consensus.cc:504] T 74d4351322b4423d81178462b8286342 P 30acad1886794455a2b5617ba6da9aa1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } }
I20240503 13:55:47.601753 9424 leader_election.cc:288] T a9cea7e1421c43fb85282e212e39715d P a7346c212aa54138970650af7f721d0e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 13:55:47.601831 9428 raft_consensus.cc:2781] T a9cea7e1421c43fb85282e212e39715d P a7346c212aa54138970650af7f721d0e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:55:47.601841 9425 leader_election.cc:302] T 74d4351322b4423d81178462b8286342 P 30acad1886794455a2b5617ba6da9aa1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30acad1886794455a2b5617ba6da9aa1; no voters:
I20240503 13:55:47.602025 9428 raft_consensus.cc:686] T a9cea7e1421c43fb85282e212e39715d P a7346c212aa54138970650af7f721d0e [term 1 LEADER]: Becoming Leader. State: Replica: a7346c212aa54138970650af7f721d0e, State: Running, Role: LEADER
I20240503 13:55:47.602139 9425 leader_election.cc:288] T 74d4351322b4423d81178462b8286342 P 30acad1886794455a2b5617ba6da9aa1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 13:55:47.602305 9428 consensus_queue.cc:237] T a9cea7e1421c43fb85282e212e39715d P a7346c212aa54138970650af7f721d0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } }
I20240503 13:55:47.602587 9424 ts_tablet_manager.cc:1398] T a9cea7e1421c43fb85282e212e39715d P a7346c212aa54138970650af7f721d0e: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20240503 13:55:47.602813 9424 tablet_bootstrap.cc:492] T 30ab7f35240e47ffb4c95fc56dc48fd3 P a7346c212aa54138970650af7f721d0e: Bootstrap starting.
I20240503 13:55:47.602877 9425 ts_tablet_manager.cc:1398] T 74d4351322b4423d81178462b8286342 P 30acad1886794455a2b5617ba6da9aa1: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240503 13:55:47.603332 9431 raft_consensus.cc:2781] T 74d4351322b4423d81178462b8286342 P 30acad1886794455a2b5617ba6da9aa1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:55:47.603524 9431 raft_consensus.cc:686] T 74d4351322b4423d81178462b8286342 P 30acad1886794455a2b5617ba6da9aa1 [term 1 LEADER]: Becoming Leader. State: Replica: 30acad1886794455a2b5617ba6da9aa1, State: Running, Role: LEADER
I20240503 13:55:47.603767 9424 tablet_bootstrap.cc:654] T 30ab7f35240e47ffb4c95fc56dc48fd3 P a7346c212aa54138970650af7f721d0e: Neither blocks nor log segments found. Creating new log.
I20240503 13:55:47.603781 9431 consensus_queue.cc:237] T 74d4351322b4423d81178462b8286342 P 30acad1886794455a2b5617ba6da9aa1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } }
I20240503 13:55:47.606254 8414 catalog_manager.cc:5484] T 74d4351322b4423d81178462b8286342 P 30acad1886794455a2b5617ba6da9aa1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30acad1886794455a2b5617ba6da9aa1 (127.6.98.66). New cstate: current_term: 1 leader_uuid: "30acad1886794455a2b5617ba6da9aa1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } } }
I20240503 13:55:47.606720 9424 tablet_bootstrap.cc:492] T 30ab7f35240e47ffb4c95fc56dc48fd3 P a7346c212aa54138970650af7f721d0e: No bootstrap required, opened a new log
I20240503 13:55:47.606802 9424 ts_tablet_manager.cc:1367] T 30ab7f35240e47ffb4c95fc56dc48fd3 P a7346c212aa54138970650af7f721d0e: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 13:55:47.606254 8409 catalog_manager.cc:5484] T a9cea7e1421c43fb85282e212e39715d P a7346c212aa54138970650af7f721d0e reported cstate change: term changed from 0 to 1, leader changed from <none> to a7346c212aa54138970650af7f721d0e (127.6.98.65). New cstate: current_term: 1 leader_uuid: "a7346c212aa54138970650af7f721d0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } } }
I20240503 13:55:47.607350 9424 raft_consensus.cc:348] T 30ab7f35240e47ffb4c95fc56dc48fd3 P a7346c212aa54138970650af7f721d0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } }
I20240503 13:55:47.607463 9424 raft_consensus.cc:374] T 30ab7f35240e47ffb4c95fc56dc48fd3 P a7346c212aa54138970650af7f721d0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:55:47.607501 9424 raft_consensus.cc:729] T 30ab7f35240e47ffb4c95fc56dc48fd3 P a7346c212aa54138970650af7f721d0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7346c212aa54138970650af7f721d0e, State: Initialized, Role: FOLLOWER
I20240503 13:55:47.607614 9424 consensus_queue.cc:260] T 30ab7f35240e47ffb4c95fc56dc48fd3 P a7346c212aa54138970650af7f721d0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } }
I20240503 13:55:47.607697 9424 raft_consensus.cc:388] T 30ab7f35240e47ffb4c95fc56dc48fd3 P a7346c212aa54138970650af7f721d0e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:55:47.607726 9424 raft_consensus.cc:482] T 30ab7f35240e47ffb4c95fc56dc48fd3 P a7346c212aa54138970650af7f721d0e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:55:47.607762 9424 raft_consensus.cc:3037] T 30ab7f35240e47ffb4c95fc56dc48fd3 P a7346c212aa54138970650af7f721d0e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:55:47.608664 9424 raft_consensus.cc:504] T 30ab7f35240e47ffb4c95fc56dc48fd3 P a7346c212aa54138970650af7f721d0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } }
I20240503 13:55:47.608783 9424 leader_election.cc:302] T 30ab7f35240e47ffb4c95fc56dc48fd3 P a7346c212aa54138970650af7f721d0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7346c212aa54138970650af7f721d0e; no voters:
I20240503 13:55:47.608872 9424 leader_election.cc:288] T 30ab7f35240e47ffb4c95fc56dc48fd3 P a7346c212aa54138970650af7f721d0e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 13:55:47.608920 9428 raft_consensus.cc:2781] T 30ab7f35240e47ffb4c95fc56dc48fd3 P a7346c212aa54138970650af7f721d0e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:55:47.609022 9428 raft_consensus.cc:686] T 30ab7f35240e47ffb4c95fc56dc48fd3 P a7346c212aa54138970650af7f721d0e [term 1 LEADER]: Becoming Leader. State: Replica: a7346c212aa54138970650af7f721d0e, State: Running, Role: LEADER
I20240503 13:55:47.609076 9424 ts_tablet_manager.cc:1398] T 30ab7f35240e47ffb4c95fc56dc48fd3 P a7346c212aa54138970650af7f721d0e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:55:47.609133 9428 consensus_queue.cc:237] T 30ab7f35240e47ffb4c95fc56dc48fd3 P a7346c212aa54138970650af7f721d0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } }
I20240503 13:55:47.610502 8409 catalog_manager.cc:5484] T 30ab7f35240e47ffb4c95fc56dc48fd3 P a7346c212aa54138970650af7f721d0e reported cstate change: term changed from 0 to 1, leader changed from <none> to a7346c212aa54138970650af7f721d0e (127.6.98.65). New cstate: current_term: 1 leader_uuid: "a7346c212aa54138970650af7f721d0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } } }
W20240503 13:55:47.726271 9441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:55:47.726542 9441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:55:47.729656 9441 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
INSERT report
rows total: 2000
time total: 79.3676 ms
time per row: 0.0396838 ms
I20240503 13:55:47.850262 8414 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45454:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:55:47.856046 8535 tablet_service.cc:1467] Processing CreateTablet for tablet 6ad697cafa9c42fcbcf0d7b22fbcb6b4 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3b9a448f00ad4aa5b759a21ca4c91628]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:55:47.856046 8666 tablet_service.cc:1467] Processing CreateTablet for tablet fbadee5b855b49dca94cc11e67bb1d21 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3b9a448f00ad4aa5b759a21ca4c91628]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:55:47.856403 8535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ad697cafa9c42fcbcf0d7b22fbcb6b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:55:47.856410 8666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbadee5b855b49dca94cc11e67bb1d21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:55:47.858641 9424 tablet_bootstrap.cc:492] T 6ad697cafa9c42fcbcf0d7b22fbcb6b4 P a7346c212aa54138970650af7f721d0e: Bootstrap starting.
I20240503 13:55:47.858896 8667 tablet_service.cc:1467] Processing CreateTablet for tablet 13c5e1f8f25345cab11a6944eae35f5a (DEFAULT_TABLE table=rebalance_test_table_1 [id=3b9a448f00ad4aa5b759a21ca4c91628]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:55:47.859140 8667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13c5e1f8f25345cab11a6944eae35f5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:55:47.859511 9424 tablet_bootstrap.cc:654] T 6ad697cafa9c42fcbcf0d7b22fbcb6b4 P a7346c212aa54138970650af7f721d0e: Neither blocks nor log segments found. Creating new log.
I20240503 13:55:47.860586 9424 tablet_bootstrap.cc:492] T 6ad697cafa9c42fcbcf0d7b22fbcb6b4 P a7346c212aa54138970650af7f721d0e: No bootstrap required, opened a new log
I20240503 13:55:47.860683 9424 ts_tablet_manager.cc:1367] T 6ad697cafa9c42fcbcf0d7b22fbcb6b4 P a7346c212aa54138970650af7f721d0e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:55:47.860935 9425 tablet_bootstrap.cc:492] T fbadee5b855b49dca94cc11e67bb1d21 P 30acad1886794455a2b5617ba6da9aa1: Bootstrap starting.
I20240503 13:55:47.861164 9424 raft_consensus.cc:348] T 6ad697cafa9c42fcbcf0d7b22fbcb6b4 P a7346c212aa54138970650af7f721d0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } }
I20240503 13:55:47.861272 9424 raft_consensus.cc:374] T 6ad697cafa9c42fcbcf0d7b22fbcb6b4 P a7346c212aa54138970650af7f721d0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:55:47.861305 9424 raft_consensus.cc:729] T 6ad697cafa9c42fcbcf0d7b22fbcb6b4 P a7346c212aa54138970650af7f721d0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7346c212aa54138970650af7f721d0e, State: Initialized, Role: FOLLOWER
I20240503 13:55:47.861431 9424 consensus_queue.cc:260] T 6ad697cafa9c42fcbcf0d7b22fbcb6b4 P a7346c212aa54138970650af7f721d0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } }
I20240503 13:55:47.861521 9424 raft_consensus.cc:388] T 6ad697cafa9c42fcbcf0d7b22fbcb6b4 P a7346c212aa54138970650af7f721d0e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:55:47.861564 9424 raft_consensus.cc:482] T 6ad697cafa9c42fcbcf0d7b22fbcb6b4 P a7346c212aa54138970650af7f721d0e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:55:47.861605 9424 raft_consensus.cc:3037] T 6ad697cafa9c42fcbcf0d7b22fbcb6b4 P a7346c212aa54138970650af7f721d0e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:55:47.861907 9425 tablet_bootstrap.cc:654] T fbadee5b855b49dca94cc11e67bb1d21 P 30acad1886794455a2b5617ba6da9aa1: Neither blocks nor log segments found. Creating new log.
I20240503 13:55:47.862268 9424 raft_consensus.cc:504] T 6ad697cafa9c42fcbcf0d7b22fbcb6b4 P a7346c212aa54138970650af7f721d0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } }
I20240503 13:55:47.862388 9424 leader_election.cc:302] T 6ad697cafa9c42fcbcf0d7b22fbcb6b4 P a7346c212aa54138970650af7f721d0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7346c212aa54138970650af7f721d0e; no voters:
I20240503 13:55:47.862476 9424 leader_election.cc:288] T 6ad697cafa9c42fcbcf0d7b22fbcb6b4 P a7346c212aa54138970650af7f721d0e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 13:55:47.862536 9429 raft_consensus.cc:2781] T 6ad697cafa9c42fcbcf0d7b22fbcb6b4 P a7346c212aa54138970650af7f721d0e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:55:47.862615 9429 raft_consensus.cc:686] T 6ad697cafa9c42fcbcf0d7b22fbcb6b4 P a7346c212aa54138970650af7f721d0e [term 1 LEADER]: Becoming Leader. State: Replica: a7346c212aa54138970650af7f721d0e, State: Running, Role: LEADER
I20240503 13:55:47.862800 9424 ts_tablet_manager.cc:1398] T 6ad697cafa9c42fcbcf0d7b22fbcb6b4 P a7346c212aa54138970650af7f721d0e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:55:47.862800 9429 consensus_queue.cc:237] T 6ad697cafa9c42fcbcf0d7b22fbcb6b4 P a7346c212aa54138970650af7f721d0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } }
I20240503 13:55:47.863231 9425 tablet_bootstrap.cc:492] T fbadee5b855b49dca94cc11e67bb1d21 P 30acad1886794455a2b5617ba6da9aa1: No bootstrap required, opened a new log
I20240503 13:55:47.863330 9425 ts_tablet_manager.cc:1367] T fbadee5b855b49dca94cc11e67bb1d21 P 30acad1886794455a2b5617ba6da9aa1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:55:47.863924 9425 raft_consensus.cc:348] T fbadee5b855b49dca94cc11e67bb1d21 P 30acad1886794455a2b5617ba6da9aa1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } }
I20240503 13:55:47.864024 9425 raft_consensus.cc:374] T fbadee5b855b49dca94cc11e67bb1d21 P 30acad1886794455a2b5617ba6da9aa1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:55:47.864063 9425 raft_consensus.cc:729] T fbadee5b855b49dca94cc11e67bb1d21 P 30acad1886794455a2b5617ba6da9aa1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30acad1886794455a2b5617ba6da9aa1, State: Initialized, Role: FOLLOWER
I20240503 13:55:47.864194 9425 consensus_queue.cc:260] T fbadee5b855b49dca94cc11e67bb1d21 P 30acad1886794455a2b5617ba6da9aa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } }
I20240503 13:55:47.864279 9425 raft_consensus.cc:388] T fbadee5b855b49dca94cc11e67bb1d21 P 30acad1886794455a2b5617ba6da9aa1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:55:47.864326 9425 raft_consensus.cc:482] T fbadee5b855b49dca94cc11e67bb1d21 P 30acad1886794455a2b5617ba6da9aa1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:55:47.864369 9425 raft_consensus.cc:3037] T fbadee5b855b49dca94cc11e67bb1d21 P 30acad1886794455a2b5617ba6da9aa1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:55:47.864400 8414 catalog_manager.cc:5484] T 6ad697cafa9c42fcbcf0d7b22fbcb6b4 P a7346c212aa54138970650af7f721d0e reported cstate change: term changed from 0 to 1, leader changed from <none> to a7346c212aa54138970650af7f721d0e (127.6.98.65). New cstate: current_term: 1 leader_uuid: "a7346c212aa54138970650af7f721d0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } } }
I20240503 13:55:47.865270 9425 raft_consensus.cc:504] T fbadee5b855b49dca94cc11e67bb1d21 P 30acad1886794455a2b5617ba6da9aa1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } }
I20240503 13:55:47.865485 9425 leader_election.cc:302] T fbadee5b855b49dca94cc11e67bb1d21 P 30acad1886794455a2b5617ba6da9aa1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30acad1886794455a2b5617ba6da9aa1; no voters:
I20240503 13:55:47.865594 9425 leader_election.cc:288] T fbadee5b855b49dca94cc11e67bb1d21 P 30acad1886794455a2b5617ba6da9aa1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 13:55:47.865716 9432 raft_consensus.cc:2781] T fbadee5b855b49dca94cc11e67bb1d21 P 30acad1886794455a2b5617ba6da9aa1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:55:47.865801 9432 raft_consensus.cc:686] T fbadee5b855b49dca94cc11e67bb1d21 P 30acad1886794455a2b5617ba6da9aa1 [term 1 LEADER]: Becoming Leader. State: Replica: 30acad1886794455a2b5617ba6da9aa1, State: Running, Role: LEADER
I20240503 13:55:47.865808 9425 ts_tablet_manager.cc:1398] T fbadee5b855b49dca94cc11e67bb1d21 P 30acad1886794455a2b5617ba6da9aa1: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 13:55:47.865922 9432 consensus_queue.cc:237] T fbadee5b855b49dca94cc11e67bb1d21 P 30acad1886794455a2b5617ba6da9aa1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } }
I20240503 13:55:47.865974 9425 tablet_bootstrap.cc:492] T 13c5e1f8f25345cab11a6944eae35f5a P 30acad1886794455a2b5617ba6da9aa1: Bootstrap starting.
I20240503 13:55:47.867003 9425 tablet_bootstrap.cc:654] T 13c5e1f8f25345cab11a6944eae35f5a P 30acad1886794455a2b5617ba6da9aa1: Neither blocks nor log segments found. Creating new log.
I20240503 13:55:47.867873 8409 catalog_manager.cc:5484] T fbadee5b855b49dca94cc11e67bb1d21 P 30acad1886794455a2b5617ba6da9aa1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30acad1886794455a2b5617ba6da9aa1 (127.6.98.66). New cstate: current_term: 1 leader_uuid: "30acad1886794455a2b5617ba6da9aa1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } } }
I20240503 13:55:47.869521 9425 tablet_bootstrap.cc:492] T 13c5e1f8f25345cab11a6944eae35f5a P 30acad1886794455a2b5617ba6da9aa1: No bootstrap required, opened a new log
I20240503 13:55:47.869617 9425 ts_tablet_manager.cc:1367] T 13c5e1f8f25345cab11a6944eae35f5a P 30acad1886794455a2b5617ba6da9aa1: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240503 13:55:47.870066 9425 raft_consensus.cc:348] T 13c5e1f8f25345cab11a6944eae35f5a P 30acad1886794455a2b5617ba6da9aa1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } }
I20240503 13:55:47.870151 9425 raft_consensus.cc:374] T 13c5e1f8f25345cab11a6944eae35f5a P 30acad1886794455a2b5617ba6da9aa1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:55:47.870180 9425 raft_consensus.cc:729] T 13c5e1f8f25345cab11a6944eae35f5a P 30acad1886794455a2b5617ba6da9aa1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30acad1886794455a2b5617ba6da9aa1, State: Initialized, Role: FOLLOWER
I20240503 13:55:47.870292 9425 consensus_queue.cc:260] T 13c5e1f8f25345cab11a6944eae35f5a P 30acad1886794455a2b5617ba6da9aa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } }
I20240503 13:55:47.870358 9425 raft_consensus.cc:388] T 13c5e1f8f25345cab11a6944eae35f5a P 30acad1886794455a2b5617ba6da9aa1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:55:47.870395 9425 raft_consensus.cc:482] T 13c5e1f8f25345cab11a6944eae35f5a P 30acad1886794455a2b5617ba6da9aa1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:55:47.870440 9425 raft_consensus.cc:3037] T 13c5e1f8f25345cab11a6944eae35f5a P 30acad1886794455a2b5617ba6da9aa1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:55:47.871271 9425 raft_consensus.cc:504] T 13c5e1f8f25345cab11a6944eae35f5a P 30acad1886794455a2b5617ba6da9aa1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } }
I20240503 13:55:47.871387 9425 leader_election.cc:302] T 13c5e1f8f25345cab11a6944eae35f5a P 30acad1886794455a2b5617ba6da9aa1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30acad1886794455a2b5617ba6da9aa1; no voters:
I20240503 13:55:47.871474 9425 leader_election.cc:288] T 13c5e1f8f25345cab11a6944eae35f5a P 30acad1886794455a2b5617ba6da9aa1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 13:55:47.871548 9432 raft_consensus.cc:2781] T 13c5e1f8f25345cab11a6944eae35f5a P 30acad1886794455a2b5617ba6da9aa1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:55:47.871639 9432 raft_consensus.cc:686] T 13c5e1f8f25345cab11a6944eae35f5a P 30acad1886794455a2b5617ba6da9aa1 [term 1 LEADER]: Becoming Leader. State: Replica: 30acad1886794455a2b5617ba6da9aa1, State: Running, Role: LEADER
I20240503 13:55:47.871695 9425 ts_tablet_manager.cc:1398] T 13c5e1f8f25345cab11a6944eae35f5a P 30acad1886794455a2b5617ba6da9aa1: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 13:55:47.871737 9432 consensus_queue.cc:237] T 13c5e1f8f25345cab11a6944eae35f5a P 30acad1886794455a2b5617ba6da9aa1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } }
I20240503 13:55:47.872890 8414 catalog_manager.cc:5484] T 13c5e1f8f25345cab11a6944eae35f5a P 30acad1886794455a2b5617ba6da9aa1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30acad1886794455a2b5617ba6da9aa1 (127.6.98.66). New cstate: current_term: 1 leader_uuid: "30acad1886794455a2b5617ba6da9aa1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } } }
W20240503 13:55:47.984206 9483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:55:47.984462 9483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:55:47.987649 9483 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
INSERT report
rows total: 2000
time total: 67.4635 ms
time per row: 0.0337318 ms
I20240503 13:55:48.096923 8410 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45454:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:55:48.102738 8535 tablet_service.cc:1467] Processing CreateTablet for tablet 093b4bbf68574f39aa3128baa4f9fc9a (DEFAULT_TABLE table=rebalance_test_table_2 [id=9453dfecf398486c84bd44eb1a914dc7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:55:48.102865 8667 tablet_service.cc:1467] Processing CreateTablet for tablet f8f620ccefff4fa580ad1a1d09c00b59 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9453dfecf398486c84bd44eb1a914dc7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:55:48.103071 8535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 093b4bbf68574f39aa3128baa4f9fc9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:55:48.103119 8536 tablet_service.cc:1467] Processing CreateTablet for tablet 8316d131c9ec41e39341f2c1a3c79722 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9453dfecf398486c84bd44eb1a914dc7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:55:48.103200 8667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8f620ccefff4fa580ad1a1d09c00b59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:55:48.103447 8536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8316d131c9ec41e39341f2c1a3c79722. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:55:48.104985 9424 tablet_bootstrap.cc:492] T 093b4bbf68574f39aa3128baa4f9fc9a P a7346c212aa54138970650af7f721d0e: Bootstrap starting.
I20240503 13:55:48.105319 9425 tablet_bootstrap.cc:492] T f8f620ccefff4fa580ad1a1d09c00b59 P 30acad1886794455a2b5617ba6da9aa1: Bootstrap starting.
I20240503 13:55:48.105854 9424 tablet_bootstrap.cc:654] T 093b4bbf68574f39aa3128baa4f9fc9a P a7346c212aa54138970650af7f721d0e: Neither blocks nor log segments found. Creating new log.
I20240503 13:55:48.106110 9425 tablet_bootstrap.cc:654] T f8f620ccefff4fa580ad1a1d09c00b59 P 30acad1886794455a2b5617ba6da9aa1: Neither blocks nor log segments found. Creating new log.
I20240503 13:55:48.107096 9424 tablet_bootstrap.cc:492] T 093b4bbf68574f39aa3128baa4f9fc9a P a7346c212aa54138970650af7f721d0e: No bootstrap required, opened a new log
I20240503 13:55:48.107146 9425 tablet_bootstrap.cc:492] T f8f620ccefff4fa580ad1a1d09c00b59 P 30acad1886794455a2b5617ba6da9aa1: No bootstrap required, opened a new log
I20240503 13:55:48.107178 9424 ts_tablet_manager.cc:1367] T 093b4bbf68574f39aa3128baa4f9fc9a P a7346c212aa54138970650af7f721d0e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:55:48.107211 9425 ts_tablet_manager.cc:1367] T f8f620ccefff4fa580ad1a1d09c00b59 P 30acad1886794455a2b5617ba6da9aa1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:55:48.107687 9425 raft_consensus.cc:348] T f8f620ccefff4fa580ad1a1d09c00b59 P 30acad1886794455a2b5617ba6da9aa1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } }
I20240503 13:55:48.107718 9424 raft_consensus.cc:348] T 093b4bbf68574f39aa3128baa4f9fc9a P a7346c212aa54138970650af7f721d0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } }
I20240503 13:55:48.107784 9425 raft_consensus.cc:374] T f8f620ccefff4fa580ad1a1d09c00b59 P 30acad1886794455a2b5617ba6da9aa1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:55:48.107806 9425 raft_consensus.cc:729] T f8f620ccefff4fa580ad1a1d09c00b59 P 30acad1886794455a2b5617ba6da9aa1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30acad1886794455a2b5617ba6da9aa1, State: Initialized, Role: FOLLOWER
I20240503 13:55:48.107805 9424 raft_consensus.cc:374] T 093b4bbf68574f39aa3128baa4f9fc9a P a7346c212aa54138970650af7f721d0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:55:48.107836 9424 raft_consensus.cc:729] T 093b4bbf68574f39aa3128baa4f9fc9a P a7346c212aa54138970650af7f721d0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7346c212aa54138970650af7f721d0e, State: Initialized, Role: FOLLOWER
I20240503 13:55:48.107915 9425 consensus_queue.cc:260] T f8f620ccefff4fa580ad1a1d09c00b59 P 30acad1886794455a2b5617ba6da9aa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } }
I20240503 13:55:48.107986 9425 raft_consensus.cc:388] T f8f620ccefff4fa580ad1a1d09c00b59 P 30acad1886794455a2b5617ba6da9aa1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:55:48.107961 9424 consensus_queue.cc:260] T 093b4bbf68574f39aa3128baa4f9fc9a P a7346c212aa54138970650af7f721d0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } }
I20240503 13:55:48.108021 9425 raft_consensus.cc:482] T f8f620ccefff4fa580ad1a1d09c00b59 P 30acad1886794455a2b5617ba6da9aa1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:55:48.108062 9425 raft_consensus.cc:3037] T f8f620ccefff4fa580ad1a1d09c00b59 P 30acad1886794455a2b5617ba6da9aa1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:55:48.108064 9424 raft_consensus.cc:388] T 093b4bbf68574f39aa3128baa4f9fc9a P a7346c212aa54138970650af7f721d0e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:55:48.108096 9424 raft_consensus.cc:482] T 093b4bbf68574f39aa3128baa4f9fc9a P a7346c212aa54138970650af7f721d0e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:55:48.108141 9424 raft_consensus.cc:3037] T 093b4bbf68574f39aa3128baa4f9fc9a P a7346c212aa54138970650af7f721d0e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:55:48.108873 9425 raft_consensus.cc:504] T f8f620ccefff4fa580ad1a1d09c00b59 P 30acad1886794455a2b5617ba6da9aa1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } }
I20240503 13:55:48.108959 9424 raft_consensus.cc:504] T 093b4bbf68574f39aa3128baa4f9fc9a P a7346c212aa54138970650af7f721d0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } }
I20240503 13:55:48.108999 9425 leader_election.cc:302] T f8f620ccefff4fa580ad1a1d09c00b59 P 30acad1886794455a2b5617ba6da9aa1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30acad1886794455a2b5617ba6da9aa1; no voters:
I20240503 13:55:48.109107 9425 leader_election.cc:288] T f8f620ccefff4fa580ad1a1d09c00b59 P 30acad1886794455a2b5617ba6da9aa1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 13:55:48.109097 9424 leader_election.cc:302] T 093b4bbf68574f39aa3128baa4f9fc9a P a7346c212aa54138970650af7f721d0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7346c212aa54138970650af7f721d0e; no voters:
I20240503 13:55:48.109170 9432 raft_consensus.cc:2781] T f8f620ccefff4fa580ad1a1d09c00b59 P 30acad1886794455a2b5617ba6da9aa1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:55:48.109262 9432 raft_consensus.cc:686] T f8f620ccefff4fa580ad1a1d09c00b59 P 30acad1886794455a2b5617ba6da9aa1 [term 1 LEADER]: Becoming Leader. State: Replica: 30acad1886794455a2b5617ba6da9aa1, State: Running, Role: LEADER
I20240503 13:55:48.109369 9432 consensus_queue.cc:237] T f8f620ccefff4fa580ad1a1d09c00b59 P 30acad1886794455a2b5617ba6da9aa1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } }
I20240503 13:55:48.109409 9425 ts_tablet_manager.cc:1398] T f8f620ccefff4fa580ad1a1d09c00b59 P 30acad1886794455a2b5617ba6da9aa1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:55:48.109694 9424 leader_election.cc:288] T 093b4bbf68574f39aa3128baa4f9fc9a P a7346c212aa54138970650af7f721d0e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 13:55:48.109745 9429 raft_consensus.cc:2781] T 093b4bbf68574f39aa3128baa4f9fc9a P a7346c212aa54138970650af7f721d0e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:55:48.109838 9429 raft_consensus.cc:686] T 093b4bbf68574f39aa3128baa4f9fc9a P a7346c212aa54138970650af7f721d0e [term 1 LEADER]: Becoming Leader. State: Replica: a7346c212aa54138970650af7f721d0e, State: Running, Role: LEADER
I20240503 13:55:48.109962 9424 ts_tablet_manager.cc:1398] T 093b4bbf68574f39aa3128baa4f9fc9a P a7346c212aa54138970650af7f721d0e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:55:48.109948 9429 consensus_queue.cc:237] T 093b4bbf68574f39aa3128baa4f9fc9a P a7346c212aa54138970650af7f721d0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } }
I20240503 13:55:48.110098 9424 tablet_bootstrap.cc:492] T 8316d131c9ec41e39341f2c1a3c79722 P a7346c212aa54138970650af7f721d0e: Bootstrap starting.
I20240503 13:55:48.110785 8410 catalog_manager.cc:5484] T f8f620ccefff4fa580ad1a1d09c00b59 P 30acad1886794455a2b5617ba6da9aa1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30acad1886794455a2b5617ba6da9aa1 (127.6.98.66). New cstate: current_term: 1 leader_uuid: "30acad1886794455a2b5617ba6da9aa1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } } }
I20240503 13:55:48.111059 9424 tablet_bootstrap.cc:654] T 8316d131c9ec41e39341f2c1a3c79722 P a7346c212aa54138970650af7f721d0e: Neither blocks nor log segments found. Creating new log.
I20240503 13:55:48.111146 8406 catalog_manager.cc:5484] T 093b4bbf68574f39aa3128baa4f9fc9a P a7346c212aa54138970650af7f721d0e reported cstate change: term changed from 0 to 1, leader changed from <none> to a7346c212aa54138970650af7f721d0e (127.6.98.65). New cstate: current_term: 1 leader_uuid: "a7346c212aa54138970650af7f721d0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } } }
I20240503 13:55:48.114078 9424 tablet_bootstrap.cc:492] T 8316d131c9ec41e39341f2c1a3c79722 P a7346c212aa54138970650af7f721d0e: No bootstrap required, opened a new log
I20240503 13:55:48.114151 9424 ts_tablet_manager.cc:1367] T 8316d131c9ec41e39341f2c1a3c79722 P a7346c212aa54138970650af7f721d0e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:55:48.114537 9424 raft_consensus.cc:348] T 8316d131c9ec41e39341f2c1a3c79722 P a7346c212aa54138970650af7f721d0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } }
I20240503 13:55:48.114617 9424 raft_consensus.cc:374] T 8316d131c9ec41e39341f2c1a3c79722 P a7346c212aa54138970650af7f721d0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:55:48.114645 9424 raft_consensus.cc:729] T 8316d131c9ec41e39341f2c1a3c79722 P a7346c212aa54138970650af7f721d0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7346c212aa54138970650af7f721d0e, State: Initialized, Role: FOLLOWER
I20240503 13:55:48.114753 9424 consensus_queue.cc:260] T 8316d131c9ec41e39341f2c1a3c79722 P a7346c212aa54138970650af7f721d0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } }
I20240503 13:55:48.114812 9424 raft_consensus.cc:388] T 8316d131c9ec41e39341f2c1a3c79722 P a7346c212aa54138970650af7f721d0e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:55:48.114851 9424 raft_consensus.cc:482] T 8316d131c9ec41e39341f2c1a3c79722 P a7346c212aa54138970650af7f721d0e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:55:48.114895 9424 raft_consensus.cc:3037] T 8316d131c9ec41e39341f2c1a3c79722 P a7346c212aa54138970650af7f721d0e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:55:48.115536 9424 raft_consensus.cc:504] T 8316d131c9ec41e39341f2c1a3c79722 P a7346c212aa54138970650af7f721d0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } }
I20240503 13:55:48.115653 9424 leader_election.cc:302] T 8316d131c9ec41e39341f2c1a3c79722 P a7346c212aa54138970650af7f721d0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7346c212aa54138970650af7f721d0e; no voters:
I20240503 13:55:48.115739 9424 leader_election.cc:288] T 8316d131c9ec41e39341f2c1a3c79722 P a7346c212aa54138970650af7f721d0e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 13:55:48.115834 9429 raft_consensus.cc:2781] T 8316d131c9ec41e39341f2c1a3c79722 P a7346c212aa54138970650af7f721d0e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:55:48.115921 9424 ts_tablet_manager.cc:1398] T 8316d131c9ec41e39341f2c1a3c79722 P a7346c212aa54138970650af7f721d0e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 13:55:48.115931 9429 raft_consensus.cc:686] T 8316d131c9ec41e39341f2c1a3c79722 P a7346c212aa54138970650af7f721d0e [term 1 LEADER]: Becoming Leader. State: Replica: a7346c212aa54138970650af7f721d0e, State: Running, Role: LEADER
I20240503 13:55:48.116144 9429 consensus_queue.cc:237] T 8316d131c9ec41e39341f2c1a3c79722 P a7346c212aa54138970650af7f721d0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } }
I20240503 13:55:48.117565 8406 catalog_manager.cc:5484] T 8316d131c9ec41e39341f2c1a3c79722 P a7346c212aa54138970650af7f721d0e reported cstate change: term changed from 0 to 1, leader changed from <none> to a7346c212aa54138970650af7f721d0e (127.6.98.65). New cstate: current_term: 1 leader_uuid: "a7346c212aa54138970650af7f721d0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } } }
W20240503 13:55:48.228458 9519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:55:48.228742 9519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:55:48.231797 9519 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
INSERT report
rows total: 2000
time total: 58.5091 ms
time per row: 0.0292545 ms
I20240503 13:55:48.332330 8408 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45454:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:55:48.337992 8667 tablet_service.cc:1467] Processing CreateTablet for tablet f1d7cdc915b3484fbfbc8d4e5ae952b5 (DEFAULT_TABLE table=rebalance_test_table_3 [id=3b610e4dbe9a4185a49772042aaa4f55]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:55:48.338006 8536 tablet_service.cc:1467] Processing CreateTablet for tablet b77257c31b6246b7a2ed06eb58a7736b (DEFAULT_TABLE table=rebalance_test_table_3 [id=3b610e4dbe9a4185a49772042aaa4f55]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:55:48.337992 8666 tablet_service.cc:1467] Processing CreateTablet for tablet 9864208496e248fdaf0ae116f3dbae6f (DEFAULT_TABLE table=rebalance_test_table_3 [id=3b610e4dbe9a4185a49772042aaa4f55]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:55:48.338318 8536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b77257c31b6246b7a2ed06eb58a7736b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:55:48.338335 8667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1d7cdc915b3484fbfbc8d4e5ae952b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:55:48.338495 8666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9864208496e248fdaf0ae116f3dbae6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:55:48.340243 9425 tablet_bootstrap.cc:492] T f1d7cdc915b3484fbfbc8d4e5ae952b5 P 30acad1886794455a2b5617ba6da9aa1: Bootstrap starting.
I20240503 13:55:48.340417 9424 tablet_bootstrap.cc:492] T b77257c31b6246b7a2ed06eb58a7736b P a7346c212aa54138970650af7f721d0e: Bootstrap starting.
I20240503 13:55:48.341253 9424 tablet_bootstrap.cc:654] T b77257c31b6246b7a2ed06eb58a7736b P a7346c212aa54138970650af7f721d0e: Neither blocks nor log segments found. Creating new log.
I20240503 13:55:48.341262 9425 tablet_bootstrap.cc:654] T f1d7cdc915b3484fbfbc8d4e5ae952b5 P 30acad1886794455a2b5617ba6da9aa1: Neither blocks nor log segments found. Creating new log.
I20240503 13:55:48.342394 9425 tablet_bootstrap.cc:492] T f1d7cdc915b3484fbfbc8d4e5ae952b5 P 30acad1886794455a2b5617ba6da9aa1: No bootstrap required, opened a new log
I20240503 13:55:48.342466 9425 ts_tablet_manager.cc:1367] T f1d7cdc915b3484fbfbc8d4e5ae952b5 P 30acad1886794455a2b5617ba6da9aa1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:55:48.342543 9424 tablet_bootstrap.cc:492] T b77257c31b6246b7a2ed06eb58a7736b P a7346c212aa54138970650af7f721d0e: No bootstrap required, opened a new log
I20240503 13:55:48.342619 9424 ts_tablet_manager.cc:1367] T b77257c31b6246b7a2ed06eb58a7736b P a7346c212aa54138970650af7f721d0e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:55:48.342880 9425 raft_consensus.cc:348] T f1d7cdc915b3484fbfbc8d4e5ae952b5 P 30acad1886794455a2b5617ba6da9aa1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } }
I20240503 13:55:48.342983 9425 raft_consensus.cc:374] T f1d7cdc915b3484fbfbc8d4e5ae952b5 P 30acad1886794455a2b5617ba6da9aa1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:55:48.343005 9425 raft_consensus.cc:729] T f1d7cdc915b3484fbfbc8d4e5ae952b5 P 30acad1886794455a2b5617ba6da9aa1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30acad1886794455a2b5617ba6da9aa1, State: Initialized, Role: FOLLOWER
I20240503 13:55:48.342972 9424 raft_consensus.cc:348] T b77257c31b6246b7a2ed06eb58a7736b P a7346c212aa54138970650af7f721d0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } }
I20240503 13:55:48.343073 9424 raft_consensus.cc:374] T b77257c31b6246b7a2ed06eb58a7736b P a7346c212aa54138970650af7f721d0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:55:48.343104 9424 raft_consensus.cc:729] T b77257c31b6246b7a2ed06eb58a7736b P a7346c212aa54138970650af7f721d0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7346c212aa54138970650af7f721d0e, State: Initialized, Role: FOLLOWER
I20240503 13:55:48.343112 9425 consensus_queue.cc:260] T f1d7cdc915b3484fbfbc8d4e5ae952b5 P 30acad1886794455a2b5617ba6da9aa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } }
I20240503 13:55:48.343191 9425 raft_consensus.cc:388] T f1d7cdc915b3484fbfbc8d4e5ae952b5 P 30acad1886794455a2b5617ba6da9aa1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:55:48.343226 9425 raft_consensus.cc:482] T f1d7cdc915b3484fbfbc8d4e5ae952b5 P 30acad1886794455a2b5617ba6da9aa1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:55:48.343216 9424 consensus_queue.cc:260] T b77257c31b6246b7a2ed06eb58a7736b P a7346c212aa54138970650af7f721d0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } }
I20240503 13:55:48.343273 9425 raft_consensus.cc:3037] T f1d7cdc915b3484fbfbc8d4e5ae952b5 P 30acad1886794455a2b5617ba6da9aa1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:55:48.343286 9424 raft_consensus.cc:388] T b77257c31b6246b7a2ed06eb58a7736b P a7346c212aa54138970650af7f721d0e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:55:48.343323 9424 raft_consensus.cc:482] T b77257c31b6246b7a2ed06eb58a7736b P a7346c212aa54138970650af7f721d0e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:55:48.343367 9424 raft_consensus.cc:3037] T b77257c31b6246b7a2ed06eb58a7736b P a7346c212aa54138970650af7f721d0e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:55:48.343953 9425 raft_consensus.cc:504] T f1d7cdc915b3484fbfbc8d4e5ae952b5 P 30acad1886794455a2b5617ba6da9aa1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } }
I20240503 13:55:48.344081 9425 leader_election.cc:302] T f1d7cdc915b3484fbfbc8d4e5ae952b5 P 30acad1886794455a2b5617ba6da9aa1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30acad1886794455a2b5617ba6da9aa1; no voters:
I20240503 13:55:48.344087 9424 raft_consensus.cc:504] T b77257c31b6246b7a2ed06eb58a7736b P a7346c212aa54138970650af7f721d0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } }
I20240503 13:55:48.344172 9425 leader_election.cc:288] T f1d7cdc915b3484fbfbc8d4e5ae952b5 P 30acad1886794455a2b5617ba6da9aa1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 13:55:48.344233 9431 raft_consensus.cc:2781] T f1d7cdc915b3484fbfbc8d4e5ae952b5 P 30acad1886794455a2b5617ba6da9aa1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:55:48.344203 9424 leader_election.cc:302] T b77257c31b6246b7a2ed06eb58a7736b P a7346c212aa54138970650af7f721d0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7346c212aa54138970650af7f721d0e; no voters:
I20240503 13:55:48.344412 9424 leader_election.cc:288] T b77257c31b6246b7a2ed06eb58a7736b P a7346c212aa54138970650af7f721d0e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 13:55:48.344409 9431 raft_consensus.cc:686] T f1d7cdc915b3484fbfbc8d4e5ae952b5 P 30acad1886794455a2b5617ba6da9aa1 [term 1 LEADER]: Becoming Leader. State: Replica: 30acad1886794455a2b5617ba6da9aa1, State: Running, Role: LEADER
I20240503 13:55:48.344584 9431 consensus_queue.cc:237] T f1d7cdc915b3484fbfbc8d4e5ae952b5 P 30acad1886794455a2b5617ba6da9aa1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } }
I20240503 13:55:48.344612 9430 raft_consensus.cc:2781] T b77257c31b6246b7a2ed06eb58a7736b P a7346c212aa54138970650af7f721d0e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:55:48.344696 9430 raft_consensus.cc:686] T b77257c31b6246b7a2ed06eb58a7736b P a7346c212aa54138970650af7f721d0e [term 1 LEADER]: Becoming Leader. State: Replica: a7346c212aa54138970650af7f721d0e, State: Running, Role: LEADER
I20240503 13:55:48.344745 9424 ts_tablet_manager.cc:1398] T b77257c31b6246b7a2ed06eb58a7736b P a7346c212aa54138970650af7f721d0e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:55:48.344805 9430 consensus_queue.cc:237] T b77257c31b6246b7a2ed06eb58a7736b P a7346c212aa54138970650af7f721d0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } }
I20240503 13:55:48.345875 8408 catalog_manager.cc:5484] T f1d7cdc915b3484fbfbc8d4e5ae952b5 P 30acad1886794455a2b5617ba6da9aa1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30acad1886794455a2b5617ba6da9aa1 (127.6.98.66). New cstate: current_term: 1 leader_uuid: "30acad1886794455a2b5617ba6da9aa1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } } }
I20240503 13:55:48.346007 9425 ts_tablet_manager.cc:1398] T f1d7cdc915b3484fbfbc8d4e5ae952b5 P 30acad1886794455a2b5617ba6da9aa1: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:55:48.346200 9425 tablet_bootstrap.cc:492] T 9864208496e248fdaf0ae116f3dbae6f P 30acad1886794455a2b5617ba6da9aa1: Bootstrap starting.
I20240503 13:55:48.346274 8410 catalog_manager.cc:5484] T b77257c31b6246b7a2ed06eb58a7736b P a7346c212aa54138970650af7f721d0e reported cstate change: term changed from 0 to 1, leader changed from <none> to a7346c212aa54138970650af7f721d0e (127.6.98.65). New cstate: current_term: 1 leader_uuid: "a7346c212aa54138970650af7f721d0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } } }
I20240503 13:55:48.348078 9425 tablet_bootstrap.cc:654] T 9864208496e248fdaf0ae116f3dbae6f P 30acad1886794455a2b5617ba6da9aa1: Neither blocks nor log segments found. Creating new log.
I20240503 13:55:48.350095 9425 tablet_bootstrap.cc:492] T 9864208496e248fdaf0ae116f3dbae6f P 30acad1886794455a2b5617ba6da9aa1: No bootstrap required, opened a new log
I20240503 13:55:48.350170 9425 ts_tablet_manager.cc:1367] T 9864208496e248fdaf0ae116f3dbae6f P 30acad1886794455a2b5617ba6da9aa1: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:55:48.350543 9425 raft_consensus.cc:348] T 9864208496e248fdaf0ae116f3dbae6f P 30acad1886794455a2b5617ba6da9aa1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } }
I20240503 13:55:48.350622 9425 raft_consensus.cc:374] T 9864208496e248fdaf0ae116f3dbae6f P 30acad1886794455a2b5617ba6da9aa1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:55:48.350648 9425 raft_consensus.cc:729] T 9864208496e248fdaf0ae116f3dbae6f P 30acad1886794455a2b5617ba6da9aa1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30acad1886794455a2b5617ba6da9aa1, State: Initialized, Role: FOLLOWER
I20240503 13:55:48.350764 9425 consensus_queue.cc:260] T 9864208496e248fdaf0ae116f3dbae6f P 30acad1886794455a2b5617ba6da9aa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } }
I20240503 13:55:48.350832 9425 raft_consensus.cc:388] T 9864208496e248fdaf0ae116f3dbae6f P 30acad1886794455a2b5617ba6da9aa1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:55:48.350869 9425 raft_consensus.cc:482] T 9864208496e248fdaf0ae116f3dbae6f P 30acad1886794455a2b5617ba6da9aa1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:55:48.350917 9425 raft_consensus.cc:3037] T 9864208496e248fdaf0ae116f3dbae6f P 30acad1886794455a2b5617ba6da9aa1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:55:48.351568 9425 raft_consensus.cc:504] T 9864208496e248fdaf0ae116f3dbae6f P 30acad1886794455a2b5617ba6da9aa1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } }
I20240503 13:55:48.351681 9425 leader_election.cc:302] T 9864208496e248fdaf0ae116f3dbae6f P 30acad1886794455a2b5617ba6da9aa1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30acad1886794455a2b5617ba6da9aa1; no voters:
I20240503 13:55:48.351760 9425 leader_election.cc:288] T 9864208496e248fdaf0ae116f3dbae6f P 30acad1886794455a2b5617ba6da9aa1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 13:55:48.351902 9547 raft_consensus.cc:2781] T 9864208496e248fdaf0ae116f3dbae6f P 30acad1886794455a2b5617ba6da9aa1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:55:48.351938 9425 ts_tablet_manager.cc:1398] T 9864208496e248fdaf0ae116f3dbae6f P 30acad1886794455a2b5617ba6da9aa1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:55:48.352380 9547 raft_consensus.cc:686] T 9864208496e248fdaf0ae116f3dbae6f P 30acad1886794455a2b5617ba6da9aa1 [term 1 LEADER]: Becoming Leader. State: Replica: 30acad1886794455a2b5617ba6da9aa1, State: Running, Role: LEADER
I20240503 13:55:48.352526 9547 consensus_queue.cc:237] T 9864208496e248fdaf0ae116f3dbae6f P 30acad1886794455a2b5617ba6da9aa1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } }
I20240503 13:55:48.353678 8410 catalog_manager.cc:5484] T 9864208496e248fdaf0ae116f3dbae6f P 30acad1886794455a2b5617ba6da9aa1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30acad1886794455a2b5617ba6da9aa1 (127.6.98.66). New cstate: current_term: 1 leader_uuid: "30acad1886794455a2b5617ba6da9aa1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } } }
W20240503 13:55:48.465889 9552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:55:48.466164 9552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:55:48.469353 9552 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
INSERT report
rows total: 2000
time total: 79.3732 ms
time per row: 0.0396866 ms
I20240503 13:55:48.590312 8407 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45454:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:55:48.596076 8536 tablet_service.cc:1467] Processing CreateTablet for tablet b20e8dd34a25448a88f0658028fad27a (DEFAULT_TABLE table=rebalance_test_table_4 [id=e6161850f776466d9d873af926bce637]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:55:48.596132 8535 tablet_service.cc:1467] Processing CreateTablet for tablet 922727f062fc4d6fbb7662a020bc3b65 (DEFAULT_TABLE table=rebalance_test_table_4 [id=e6161850f776466d9d873af926bce637]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:55:48.596416 8536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b20e8dd34a25448a88f0658028fad27a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:55:48.596779 8535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 922727f062fc4d6fbb7662a020bc3b65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:55:48.596724 8666 tablet_service.cc:1467] Processing CreateTablet for tablet ff18150f2a6e4a07b4ae653bf899a8d0 (DEFAULT_TABLE table=rebalance_test_table_4 [id=e6161850f776466d9d873af926bce637]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:55:48.596983 8666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff18150f2a6e4a07b4ae653bf899a8d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:55:48.598820 9424 tablet_bootstrap.cc:492] T 922727f062fc4d6fbb7662a020bc3b65 P a7346c212aa54138970650af7f721d0e: Bootstrap starting.
I20240503 13:55:48.598945 9425 tablet_bootstrap.cc:492] T ff18150f2a6e4a07b4ae653bf899a8d0 P 30acad1886794455a2b5617ba6da9aa1: Bootstrap starting.
I20240503 13:55:48.599663 9424 tablet_bootstrap.cc:654] T 922727f062fc4d6fbb7662a020bc3b65 P a7346c212aa54138970650af7f721d0e: Neither blocks nor log segments found. Creating new log.
I20240503 13:55:48.599721 9425 tablet_bootstrap.cc:654] T ff18150f2a6e4a07b4ae653bf899a8d0 P 30acad1886794455a2b5617ba6da9aa1: Neither blocks nor log segments found. Creating new log.
I20240503 13:55:48.600853 9424 tablet_bootstrap.cc:492] T 922727f062fc4d6fbb7662a020bc3b65 P a7346c212aa54138970650af7f721d0e: No bootstrap required, opened a new log
I20240503 13:55:48.600942 9424 ts_tablet_manager.cc:1367] T 922727f062fc4d6fbb7662a020bc3b65 P a7346c212aa54138970650af7f721d0e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:55:48.601147 9425 tablet_bootstrap.cc:492] T ff18150f2a6e4a07b4ae653bf899a8d0 P 30acad1886794455a2b5617ba6da9aa1: No bootstrap required, opened a new log
I20240503 13:55:48.601215 9425 ts_tablet_manager.cc:1367] T ff18150f2a6e4a07b4ae653bf899a8d0 P 30acad1886794455a2b5617ba6da9aa1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:55:48.601442 9424 raft_consensus.cc:348] T 922727f062fc4d6fbb7662a020bc3b65 P a7346c212aa54138970650af7f721d0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } }
I20240503 13:55:48.601557 9424 raft_consensus.cc:374] T 922727f062fc4d6fbb7662a020bc3b65 P a7346c212aa54138970650af7f721d0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:55:48.601590 9424 raft_consensus.cc:729] T 922727f062fc4d6fbb7662a020bc3b65 P a7346c212aa54138970650af7f721d0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7346c212aa54138970650af7f721d0e, State: Initialized, Role: FOLLOWER
I20240503 13:55:48.601725 9424 consensus_queue.cc:260] T 922727f062fc4d6fbb7662a020bc3b65 P a7346c212aa54138970650af7f721d0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } }
I20240503 13:55:48.601735 9425 raft_consensus.cc:348] T ff18150f2a6e4a07b4ae653bf899a8d0 P 30acad1886794455a2b5617ba6da9aa1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } }
I20240503 13:55:48.601804 9424 raft_consensus.cc:388] T 922727f062fc4d6fbb7662a020bc3b65 P a7346c212aa54138970650af7f721d0e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:55:48.601815 9425 raft_consensus.cc:374] T ff18150f2a6e4a07b4ae653bf899a8d0 P 30acad1886794455a2b5617ba6da9aa1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:55:48.601835 9424 raft_consensus.cc:482] T 922727f062fc4d6fbb7662a020bc3b65 P a7346c212aa54138970650af7f721d0e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:55:48.601845 9425 raft_consensus.cc:729] T ff18150f2a6e4a07b4ae653bf899a8d0 P 30acad1886794455a2b5617ba6da9aa1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30acad1886794455a2b5617ba6da9aa1, State: Initialized, Role: FOLLOWER
I20240503 13:55:48.601876 9424 raft_consensus.cc:3037] T 922727f062fc4d6fbb7662a020bc3b65 P a7346c212aa54138970650af7f721d0e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:55:48.601960 9425 consensus_queue.cc:260] T ff18150f2a6e4a07b4ae653bf899a8d0 P 30acad1886794455a2b5617ba6da9aa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } }
I20240503 13:55:48.602039 9425 raft_consensus.cc:388] T ff18150f2a6e4a07b4ae653bf899a8d0 P 30acad1886794455a2b5617ba6da9aa1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:55:48.602070 9425 raft_consensus.cc:482] T ff18150f2a6e4a07b4ae653bf899a8d0 P 30acad1886794455a2b5617ba6da9aa1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:55:48.602105 9425 raft_consensus.cc:3037] T ff18150f2a6e4a07b4ae653bf899a8d0 P 30acad1886794455a2b5617ba6da9aa1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:55:48.602818 9424 raft_consensus.cc:504] T 922727f062fc4d6fbb7662a020bc3b65 P a7346c212aa54138970650af7f721d0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } }
I20240503 13:55:48.602891 9425 raft_consensus.cc:504] T ff18150f2a6e4a07b4ae653bf899a8d0 P 30acad1886794455a2b5617ba6da9aa1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } }
I20240503 13:55:48.602947 9424 leader_election.cc:302] T 922727f062fc4d6fbb7662a020bc3b65 P a7346c212aa54138970650af7f721d0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7346c212aa54138970650af7f721d0e; no voters:
I20240503 13:55:48.602991 9425 leader_election.cc:302] T ff18150f2a6e4a07b4ae653bf899a8d0 P 30acad1886794455a2b5617ba6da9aa1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30acad1886794455a2b5617ba6da9aa1; no voters:
I20240503 13:55:48.603052 9424 leader_election.cc:288] T 922727f062fc4d6fbb7662a020bc3b65 P a7346c212aa54138970650af7f721d0e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 13:55:48.603075 9425 leader_election.cc:288] T ff18150f2a6e4a07b4ae653bf899a8d0 P 30acad1886794455a2b5617ba6da9aa1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 13:55:48.603125 9429 raft_consensus.cc:2781] T 922727f062fc4d6fbb7662a020bc3b65 P a7346c212aa54138970650af7f721d0e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:55:48.603132 9547 raft_consensus.cc:2781] T ff18150f2a6e4a07b4ae653bf899a8d0 P 30acad1886794455a2b5617ba6da9aa1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:55:48.603222 9547 raft_consensus.cc:686] T ff18150f2a6e4a07b4ae653bf899a8d0 P 30acad1886794455a2b5617ba6da9aa1 [term 1 LEADER]: Becoming Leader. State: Replica: 30acad1886794455a2b5617ba6da9aa1, State: Running, Role: LEADER
I20240503 13:55:48.603284 9425 ts_tablet_manager.cc:1398] T ff18150f2a6e4a07b4ae653bf899a8d0 P 30acad1886794455a2b5617ba6da9aa1: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 13:55:48.603336 9547 consensus_queue.cc:237] T ff18150f2a6e4a07b4ae653bf899a8d0 P 30acad1886794455a2b5617ba6da9aa1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } }
I20240503 13:55:48.603672 9429 raft_consensus.cc:686] T 922727f062fc4d6fbb7662a020bc3b65 P a7346c212aa54138970650af7f721d0e [term 1 LEADER]: Becoming Leader. State: Replica: a7346c212aa54138970650af7f721d0e, State: Running, Role: LEADER
I20240503 13:55:48.603798 9429 consensus_queue.cc:237] T 922727f062fc4d6fbb7662a020bc3b65 P a7346c212aa54138970650af7f721d0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } }
I20240503 13:55:48.604537 9424 ts_tablet_manager.cc:1398] T 922727f062fc4d6fbb7662a020bc3b65 P a7346c212aa54138970650af7f721d0e: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 13:55:48.604718 9424 tablet_bootstrap.cc:492] T b20e8dd34a25448a88f0658028fad27a P a7346c212aa54138970650af7f721d0e: Bootstrap starting.
I20240503 13:55:48.604810 8407 catalog_manager.cc:5484] T ff18150f2a6e4a07b4ae653bf899a8d0 P 30acad1886794455a2b5617ba6da9aa1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30acad1886794455a2b5617ba6da9aa1 (127.6.98.66). New cstate: current_term: 1 leader_uuid: "30acad1886794455a2b5617ba6da9aa1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } } }
I20240503 13:55:48.605018 8414 catalog_manager.cc:5484] T 922727f062fc4d6fbb7662a020bc3b65 P a7346c212aa54138970650af7f721d0e reported cstate change: term changed from 0 to 1, leader changed from <none> to a7346c212aa54138970650af7f721d0e (127.6.98.65). New cstate: current_term: 1 leader_uuid: "a7346c212aa54138970650af7f721d0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } } }
I20240503 13:55:48.605825 9424 tablet_bootstrap.cc:654] T b20e8dd34a25448a88f0658028fad27a P a7346c212aa54138970650af7f721d0e: Neither blocks nor log segments found. Creating new log.
I20240503 13:55:48.608701 9424 tablet_bootstrap.cc:492] T b20e8dd34a25448a88f0658028fad27a P a7346c212aa54138970650af7f721d0e: No bootstrap required, opened a new log
I20240503 13:55:48.608799 9424 ts_tablet_manager.cc:1367] T b20e8dd34a25448a88f0658028fad27a P a7346c212aa54138970650af7f721d0e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:55:48.609140 9424 raft_consensus.cc:348] T b20e8dd34a25448a88f0658028fad27a P a7346c212aa54138970650af7f721d0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } }
I20240503 13:55:48.609216 9424 raft_consensus.cc:374] T b20e8dd34a25448a88f0658028fad27a P a7346c212aa54138970650af7f721d0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:55:48.609236 9424 raft_consensus.cc:729] T b20e8dd34a25448a88f0658028fad27a P a7346c212aa54138970650af7f721d0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7346c212aa54138970650af7f721d0e, State: Initialized, Role: FOLLOWER
I20240503 13:55:48.609323 9424 consensus_queue.cc:260] T b20e8dd34a25448a88f0658028fad27a P a7346c212aa54138970650af7f721d0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } }
I20240503 13:55:48.609370 9424 raft_consensus.cc:388] T b20e8dd34a25448a88f0658028fad27a P a7346c212aa54138970650af7f721d0e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:55:48.609390 9424 raft_consensus.cc:482] T b20e8dd34a25448a88f0658028fad27a P a7346c212aa54138970650af7f721d0e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:55:48.609421 9424 raft_consensus.cc:3037] T b20e8dd34a25448a88f0658028fad27a P a7346c212aa54138970650af7f721d0e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:55:48.610230 9424 raft_consensus.cc:504] T b20e8dd34a25448a88f0658028fad27a P a7346c212aa54138970650af7f721d0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } }
I20240503 13:55:48.610368 9424 leader_election.cc:302] T b20e8dd34a25448a88f0658028fad27a P a7346c212aa54138970650af7f721d0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7346c212aa54138970650af7f721d0e; no voters:
I20240503 13:55:48.610469 9424 leader_election.cc:288] T b20e8dd34a25448a88f0658028fad27a P a7346c212aa54138970650af7f721d0e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 13:55:48.610517 9428 raft_consensus.cc:2781] T b20e8dd34a25448a88f0658028fad27a P a7346c212aa54138970650af7f721d0e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:55:48.610656 9428 raft_consensus.cc:686] T b20e8dd34a25448a88f0658028fad27a P a7346c212aa54138970650af7f721d0e [term 1 LEADER]: Becoming Leader. State: Replica: a7346c212aa54138970650af7f721d0e, State: Running, Role: LEADER
I20240503 13:55:48.610698 9424 ts_tablet_manager.cc:1398] T b20e8dd34a25448a88f0658028fad27a P a7346c212aa54138970650af7f721d0e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 13:55:48.610765 9428 consensus_queue.cc:237] T b20e8dd34a25448a88f0658028fad27a P a7346c212aa54138970650af7f721d0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } }
I20240503 13:55:48.611959 8407 catalog_manager.cc:5484] T b20e8dd34a25448a88f0658028fad27a P a7346c212aa54138970650af7f721d0e reported cstate change: term changed from 0 to 1, leader changed from <none> to a7346c212aa54138970650af7f721d0e (127.6.98.65). New cstate: current_term: 1 leader_uuid: "a7346c212aa54138970650af7f721d0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } } }
W20240503 13:55:48.720944 9584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:55:48.721202 9584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:55:48.724784 9584 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
INSERT report
rows total: 2000
time total: 73.3695 ms
time per row: 0.0366847 ms
I20240503 13:55:48.847343 6537 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
/tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.98.67:40171
--local_ip_for_outbound_sockets=127.6.98.67
--tserver_master_addrs=127.6.98.126:36629
--webserver_port=34021
--webserver_interface=127.6.98.67
--builtin_ntp_servers=127.6.98.84:33479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 13:55:48.960045 9613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:55:48.960283 9613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:55:48.960326 9613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:55:48.963327 9613 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 13:55:48.963392 9613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:55:48.963460 9613 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.98.67
I20240503 13:55:48.967269 9613 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.98.84:33479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.98.67:40171
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.6.98.67
--webserver_port=34021
--tserver_master_addrs=127.6.98.126:36629
--never_fsync=true
--heap_profile_path=/tmp/kudu.9613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.98.67
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:55:48.968183 9613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:55:48.969300 9613 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:55:48.977268 9621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:55:48.977581 9618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:55:48.978006 9613 server_base.cc:1034] running on GCE node
W20240503 13:55:48.978490 9619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:55:48.978893 9613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:55:48.979466 9613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:55:48.980603 9613 hybrid_clock.cc:648] HybridClock initialized: now 1714744548980587 us; error 43 us; skew 500 ppm
I20240503 13:55:48.982414 9613 webserver.cc:460] Webserver started at http://127.6.98.67:34021/ using document root <none> and password file <none>
I20240503 13:55:48.982949 9613 fs_manager.cc:359] Metadata directory not provided
I20240503 13:55:48.983028 9613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:55:48.986402 9613 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20240503 13:55:48.988273 9627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:55:48.989310 9613 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20240503 13:55:48.989419 9613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "45bdf7ba415141cd85008dca119d1cf9"
format_stamp: "Formatted at 2024-05-03 13:55:42 on dist-test-slave-zg9z"
I20240503 13:55:48.989830 9613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:55:49.008865 9613 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:55:49.009538 9613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:55:49.009698 9613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:55:49.010243 9613 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:55:49.011153 9613 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:55:49.011200 9613 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:55:49.011253 9613 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:55:49.011282 9613 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:55:49.019804 9613 rpc_server.cc:307] RPC server started. Bound to: 127.6.98.67:40171
I20240503 13:55:49.019878 9740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.98.67:40171 every 8 connection(s)
I20240503 13:55:49.020876 9613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 13:55:49.023453 6537 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu as pid 9613
I20240503 13:55:49.023900 6537 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
/tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.98.68:40653
--local_ip_for_outbound_sockets=127.6.98.68
--tserver_master_addrs=127.6.98.126:36629
--webserver_port=45725
--webserver_interface=127.6.98.68
--builtin_ntp_servers=127.6.98.84:33479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 13:55:49.030772 9741 heartbeater.cc:347] Connected to a master server at 127.6.98.126:36629
I20240503 13:55:49.031026 9741 heartbeater.cc:464] Registering TS with master...
I20240503 13:55:49.031497 9741 heartbeater.cc:511] Master 127.6.98.126:36629 requested a full tablet report, sending...
I20240503 13:55:49.032668 8408 ts_manager.cc:194] Re-registered known tserver with Master: 45bdf7ba415141cd85008dca119d1cf9 (127.6.98.67:40171)
I20240503 13:55:49.033254 8408 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.98.67:37545
W20240503 13:55:49.133788 9744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:55:49.134024 9744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:55:49.134086 9744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:55:49.137197 9744 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 13:55:49.137285 9744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:55:49.137377 9744 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.98.68
I20240503 13:55:49.141125 9744 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.98.84:33479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.98.68:40653
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.6.98.68
--webserver_port=45725
--tserver_master_addrs=127.6.98.126:36629
--never_fsync=true
--heap_profile_path=/tmp/kudu.9744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.98.68
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:55:49.142000 9744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:55:49.143086 9744 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:55:49.151059 9752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:55:49.151392 9750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:55:49.151770 9749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:55:49.152348 9744 server_base.cc:1034] running on GCE node
I20240503 13:55:49.152755 9744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:55:49.153335 9744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:55:49.154482 9744 hybrid_clock.cc:648] HybridClock initialized: now 1714744549154461 us; error 36 us; skew 500 ppm
I20240503 13:55:49.156325 9744 webserver.cc:460] Webserver started at http://127.6.98.68:45725/ using document root <none> and password file <none>
I20240503 13:55:49.156934 9744 fs_manager.cc:359] Metadata directory not provided
I20240503 13:55:49.157006 9744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:55:49.160336 9744 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20240503 13:55:49.162485 9758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:55:49.163538 9744 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:55:49.163635 9744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "91d1a791a7504f71a4785f6aefe26f1b"
format_stamp: "Formatted at 2024-05-03 13:55:43 on dist-test-slave-zg9z"
I20240503 13:55:49.164098 9744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:55:49.187659 9744 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:55:49.188287 9744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:55:49.188418 9744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:55:49.188962 9744 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:55:49.190016 9765 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
I20240503 13:55:49.193053 9744 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240503 13:55:49.193113 9744 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20240503 13:55:49.193164 9744 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240503 13:55:49.194689 9744 ts_tablet_manager.cc:590] Registered 1 tablets
I20240503 13:55:49.194729 9744 ts_tablet_manager.cc:569] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20240503 13:55:49.194972 9765 tablet_bootstrap.cc:492] T 331586b138a94590b0a06e6fec84bf4a P 91d1a791a7504f71a4785f6aefe26f1b: Bootstrap starting.
I20240503 13:55:49.204802 9744 rpc_server.cc:307] RPC server started. Bound to: 127.6.98.68:40653
I20240503 13:55:49.204859 9872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.98.68:40653 every 8 connection(s)
I20240503 13:55:49.205788 9744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 13:55:49.208773 9765 log.cc:824] T 331586b138a94590b0a06e6fec84bf4a P 91d1a791a7504f71a4785f6aefe26f1b: Log is configured to *not* fsync() on all Append() calls
I20240503 13:55:49.210660 6537 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu as pid 9744
I20240503 13:55:49.211045 6537 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
/tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.98.69:33575
--local_ip_for_outbound_sockets=127.6.98.69
--tserver_master_addrs=127.6.98.126:36629
--webserver_port=41739
--webserver_interface=127.6.98.69
--builtin_ntp_servers=127.6.98.84:33479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 13:55:49.211582 9765 tablet_bootstrap.cc:492] T 331586b138a94590b0a06e6fec84bf4a P 91d1a791a7504f71a4785f6aefe26f1b: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:55:49.211925 9765 tablet_bootstrap.cc:492] T 331586b138a94590b0a06e6fec84bf4a P 91d1a791a7504f71a4785f6aefe26f1b: Bootstrap complete.
I20240503 13:55:49.213075 9765 ts_tablet_manager.cc:1367] T 331586b138a94590b0a06e6fec84bf4a P 91d1a791a7504f71a4785f6aefe26f1b: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.010s
I20240503 13:55:49.215535 9873 heartbeater.cc:347] Connected to a master server at 127.6.98.126:36629
I20240503 13:55:49.215759 9873 heartbeater.cc:464] Registering TS with master...
I20240503 13:55:49.216275 9873 heartbeater.cc:511] Master 127.6.98.126:36629 requested a full tablet report, sending...
I20240503 13:55:49.216183 9765 raft_consensus.cc:348] T 331586b138a94590b0a06e6fec84bf4a P 91d1a791a7504f71a4785f6aefe26f1b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91d1a791a7504f71a4785f6aefe26f1b" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 40653 } }
I20240503 13:55:49.216572 9765 raft_consensus.cc:729] T 331586b138a94590b0a06e6fec84bf4a P 91d1a791a7504f71a4785f6aefe26f1b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d1a791a7504f71a4785f6aefe26f1b, State: Initialized, Role: FOLLOWER
I20240503 13:55:49.217008 9765 consensus_queue.cc:260] T 331586b138a94590b0a06e6fec84bf4a P 91d1a791a7504f71a4785f6aefe26f1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91d1a791a7504f71a4785f6aefe26f1b" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 40653 } }
I20240503 13:55:49.217131 9765 raft_consensus.cc:388] T 331586b138a94590b0a06e6fec84bf4a P 91d1a791a7504f71a4785f6aefe26f1b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:55:49.217192 9765 raft_consensus.cc:482] T 331586b138a94590b0a06e6fec84bf4a P 91d1a791a7504f71a4785f6aefe26f1b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:55:49.217286 9765 raft_consensus.cc:3037] T 331586b138a94590b0a06e6fec84bf4a P 91d1a791a7504f71a4785f6aefe26f1b [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:55:49.218783 9765 raft_consensus.cc:504] T 331586b138a94590b0a06e6fec84bf4a P 91d1a791a7504f71a4785f6aefe26f1b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91d1a791a7504f71a4785f6aefe26f1b" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 40653 } }
I20240503 13:55:49.219125 9765 leader_election.cc:302] T 331586b138a94590b0a06e6fec84bf4a P 91d1a791a7504f71a4785f6aefe26f1b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 91d1a791a7504f71a4785f6aefe26f1b; no voters:
I20240503 13:55:49.219396 9765 leader_election.cc:288] T 331586b138a94590b0a06e6fec84bf4a P 91d1a791a7504f71a4785f6aefe26f1b [CANDIDATE]: Term 2 election: Requested vote from peers
I20240503 13:55:49.219558 9878 raft_consensus.cc:2781] T 331586b138a94590b0a06e6fec84bf4a P 91d1a791a7504f71a4785f6aefe26f1b [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:55:49.219749 9878 raft_consensus.cc:686] T 331586b138a94590b0a06e6fec84bf4a P 91d1a791a7504f71a4785f6aefe26f1b [term 2 LEADER]: Becoming Leader. State: Replica: 91d1a791a7504f71a4785f6aefe26f1b, State: Running, Role: LEADER
I20240503 13:55:49.220062 9878 consensus_queue.cc:237] T 331586b138a94590b0a06e6fec84bf4a P 91d1a791a7504f71a4785f6aefe26f1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91d1a791a7504f71a4785f6aefe26f1b" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 40653 } }
I20240503 13:55:49.220237 9765 ts_tablet_manager.cc:1398] T 331586b138a94590b0a06e6fec84bf4a P 91d1a791a7504f71a4785f6aefe26f1b: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20240503 13:55:49.221238 8408 ts_manager.cc:194] Re-registered known tserver with Master: 91d1a791a7504f71a4785f6aefe26f1b (127.6.98.68:40653)
I20240503 13:55:49.222293 8408 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.98.68:44883
I20240503 13:55:49.224156 9873 heartbeater.cc:503] Master 127.6.98.126:36629 was elected leader, sending a full tablet report...
I20240503 13:55:49.225087 8408 catalog_manager.cc:5484] T 331586b138a94590b0a06e6fec84bf4a P 91d1a791a7504f71a4785f6aefe26f1b reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "91d1a791a7504f71a4785f6aefe26f1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91d1a791a7504f71a4785f6aefe26f1b" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 40653 } } }
W20240503 13:55:49.321067 9877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:55:49.321288 9877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:55:49.321336 9877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:55:49.324640 9877 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 13:55:49.324714 9877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:55:49.324800 9877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.98.69
I20240503 13:55:49.328681 9877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.98.84:33479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.98.69:33575
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.6.98.69
--webserver_port=41739
--tserver_master_addrs=127.6.98.126:36629
--never_fsync=true
--heap_profile_path=/tmp/kudu.9877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.98.69
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:55:49.329643 9877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:55:49.330762 9877 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:55:49.338713 9890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:55:49.339164 9891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:55:49.339638 9893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:55:49.340122 9877 server_base.cc:1034] running on GCE node
I20240503 13:55:49.340528 9877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:55:49.341094 9877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:55:49.342237 9877 hybrid_clock.cc:648] HybridClock initialized: now 1714744549342220 us; error 31 us; skew 500 ppm
I20240503 13:55:49.344017 9877 webserver.cc:460] Webserver started at http://127.6.98.69:41739/ using document root <none> and password file <none>
I20240503 13:55:49.344620 9877 fs_manager.cc:359] Metadata directory not provided
I20240503 13:55:49.344695 9877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:55:49.348088 9877 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 13:55:49.350232 9899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:55:49.351363 9877 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:55:49.351471 9877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "6a0591a0f0b64c418af287c96b5b47b8"
format_stamp: "Formatted at 2024-05-03 13:55:43 on dist-test-slave-zg9z"
I20240503 13:55:49.351974 9877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:55:49.377801 9877 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:55:49.378454 9877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:55:49.378592 9877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:55:49.379088 9877 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:55:49.379933 9877 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:55:49.379978 9877 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:55:49.380040 9877 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:55:49.380060 9877 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:55:49.388267 9877 rpc_server.cc:307] RPC server started. Bound to: 127.6.98.69:33575
I20240503 13:55:49.388296 10012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.98.69:33575 every 8 connection(s)
I20240503 13:55:49.389302 9877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 13:55:49.398200 10013 heartbeater.cc:347] Connected to a master server at 127.6.98.126:36629
I20240503 13:55:49.398448 10013 heartbeater.cc:464] Registering TS with master...
I20240503 13:55:49.398852 6537 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu as pid 9877
I20240503 13:55:49.399037 10013 heartbeater.cc:511] Master 127.6.98.126:36629 requested a full tablet report, sending...
I20240503 13:55:49.399277 6537 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
/tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.98.70:33391
--local_ip_for_outbound_sockets=127.6.98.70
--tserver_master_addrs=127.6.98.126:36629
--webserver_port=42253
--webserver_interface=127.6.98.70
--builtin_ntp_servers=127.6.98.84:33479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 13:55:49.400127 8408 ts_manager.cc:194] Re-registered known tserver with Master: 6a0591a0f0b64c418af287c96b5b47b8 (127.6.98.69:33575)
I20240503 13:55:49.400725 8408 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.98.69:36053
W20240503 13:55:49.509467 10016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:55:49.509704 10016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:55:49.509752 10016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:55:49.513370 10016 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 13:55:49.513455 10016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:55:49.513527 10016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.98.70
I20240503 13:55:49.517426 10016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.98.84:33479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.98.70:33391
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.6.98.70
--webserver_port=42253
--tserver_master_addrs=127.6.98.126:36629
--never_fsync=true
--heap_profile_path=/tmp/kudu.10016
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.98.70
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:55:49.518314 10016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:55:49.519384 10016 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:55:49.527215 10021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:55:49.527639 10024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:55:49.527963 10022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:55:49.528645 10016 server_base.cc:1034] running on GCE node
I20240503 13:55:49.529047 10016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:55:49.529650 10016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:55:49.530813 10016 hybrid_clock.cc:648] HybridClock initialized: now 1714744549530775 us; error 55 us; skew 500 ppm
I20240503 13:55:49.532807 10016 webserver.cc:460] Webserver started at http://127.6.98.70:42253/ using document root <none> and password file <none>
I20240503 13:55:49.533380 10016 fs_manager.cc:359] Metadata directory not provided
I20240503 13:55:49.533457 10016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:55:49.536981 10016 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 13:55:49.538853 10030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:55:49.539848 10016 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:55:49.539945 10016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "a075f746a2e04544931d75c3dc463ef3"
format_stamp: "Formatted at 2024-05-03 13:55:43 on dist-test-slave-zg9z"
I20240503 13:55:49.540392 10016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:55:49.556578 10016 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:55:49.557246 10016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:55:49.557405 10016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:55:49.558003 10016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:55:49.558925 10016 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:55:49.558974 10016 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:55:49.559012 10016 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:55:49.559026 10016 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:55:49.567615 10016 rpc_server.cc:307] RPC server started. Bound to: 127.6.98.70:33391
I20240503 13:55:49.567684 10143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.98.70:33391 every 8 connection(s)
I20240503 13:55:49.568755 10016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 13:55:49.575599 6537 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu as pid 10016
I20240503 13:55:49.576027 6537 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
/tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.98.71:39765
--local_ip_for_outbound_sockets=127.6.98.71
--tserver_master_addrs=127.6.98.126:36629
--webserver_port=39801
--webserver_interface=127.6.98.71
--builtin_ntp_servers=127.6.98.84:33479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 13:55:49.577652 10144 heartbeater.cc:347] Connected to a master server at 127.6.98.126:36629
I20240503 13:55:49.577903 10144 heartbeater.cc:464] Registering TS with master...
I20240503 13:55:49.578440 10144 heartbeater.cc:511] Master 127.6.98.126:36629 requested a full tablet report, sending...
I20240503 13:55:49.579393 8408 ts_manager.cc:194] Re-registered known tserver with Master: a075f746a2e04544931d75c3dc463ef3 (127.6.98.70:33391)
I20240503 13:55:49.580022 8408 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.98.70:51509
W20240503 13:55:49.693176 10147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:55:49.693419 10147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:55:49.693466 10147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:55:49.696606 10147 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 13:55:49.696669 10147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:55:49.696735 10147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.98.71
I20240503 13:55:49.700369 10147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.98.84:33479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.98.71:39765
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.6.98.71
--webserver_port=39801
--tserver_master_addrs=127.6.98.126:36629
--never_fsync=true
--heap_profile_path=/tmp/kudu.10147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.98.71
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:55:49.701290 10147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:55:49.702468 10147 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:55:49.710731 10153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:55:49.711050 10147 server_base.cc:1034] running on GCE node
W20240503 13:55:49.711194 10152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:55:49.711694 10155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:55:49.712128 10147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:55:49.712742 10147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:55:49.713892 10147 hybrid_clock.cc:648] HybridClock initialized: now 1714744549713877 us; error 30 us; skew 500 ppm
I20240503 13:55:49.715862 10147 webserver.cc:460] Webserver started at http://127.6.98.71:39801/ using document root <none> and password file <none>
I20240503 13:55:49.716410 10147 fs_manager.cc:359] Metadata directory not provided
I20240503 13:55:49.716478 10147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:55:49.719763 10147 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20240503 13:55:49.721751 10161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:55:49.722847 10147 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 13:55:49.722942 10147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "44f093b99a6a403f8e092124ba9fea4c"
format_stamp: "Formatted at 2024-05-03 13:55:43 on dist-test-slave-zg9z"
I20240503 13:55:49.723390 10147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:55:49.738669 10147 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:55:49.739306 10147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:55:49.739449 10147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:55:49.739974 10147 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:55:49.740908 10147 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:55:49.740962 10147 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:55:49.741005 10147 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:55:49.741025 10147 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:55:49.749322 10147 rpc_server.cc:307] RPC server started. Bound to: 127.6.98.71:39765
I20240503 13:55:49.749369 10274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.98.71:39765 every 8 connection(s)
I20240503 13:55:49.750383 10147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 13:55:49.753981 6537 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu as pid 10147
I20240503 13:55:49.760574 10275 heartbeater.cc:347] Connected to a master server at 127.6.98.126:36629
I20240503 13:55:49.760870 10275 heartbeater.cc:464] Registering TS with master...
I20240503 13:55:49.761546 10275 heartbeater.cc:511] Master 127.6.98.126:36629 requested a full tablet report, sending...
I20240503 13:55:49.762761 8408 ts_manager.cc:194] Re-registered known tserver with Master: 44f093b99a6a403f8e092124ba9fea4c (127.6.98.71:39765)
I20240503 13:55:49.763500 8408 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.98.71:55849
I20240503 13:55:49.950603 9947 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:55:49.952096 8666 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:55:49.953414 9807 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:55:49.957924 9675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:55:49.960076 8536 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:55:49.962010 10078 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:55:49.965009 10209 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:55:50.035216 9741 heartbeater.cc:503] Master 127.6.98.126:36629 was elected leader, sending a full tablet report...
I20240503 13:55:50.098762 10209 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:55:50.099654 8666 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:55:50.100124 9675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:55:50.101480 9947 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:55:50.106195 9807 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:55:50.126137 10078 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:55:50.126299 8536 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:55:50.170441 8556 consensus_queue.cc:237] T 922727f062fc4d6fbb7662a020bc3b65 P a7346c212aa54138970650af7f721d0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } } peers { permanent_uuid: "45bdf7ba415141cd85008dca119d1cf9" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 40171 } }
W20240503 13:55:50.175634 8490 consensus_peers.cc:485] T 922727f062fc4d6fbb7662a020bc3b65 P a7346c212aa54138970650af7f721d0e -> Peer 45bdf7ba415141cd85008dca119d1cf9 (127.6.98.67:40171): Couldn't send request to peer 45bdf7ba415141cd85008dca119d1cf9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 922727f062fc4d6fbb7662a020bc3b65. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:55:50.403108 10013 heartbeater.cc:503] Master 127.6.98.126:36629 was elected leader, sending a full tablet report...
I20240503 13:55:50.581998 10144 heartbeater.cc:503] Master 127.6.98.126:36629 was elected leader, sending a full tablet report...
I20240503 13:55:50.673802 10378 ts_tablet_manager.cc:902] T 922727f062fc4d6fbb7662a020bc3b65 P 45bdf7ba415141cd85008dca119d1cf9: Initiating tablet copy from peer a7346c212aa54138970650af7f721d0e (127.6.98.65:37097)
I20240503 13:55:50.674588 10378 tablet_copy_client.cc:308] T 922727f062fc4d6fbb7662a020bc3b65 P 45bdf7ba415141cd85008dca119d1cf9: tablet copy: Beginning tablet copy session from remote peer at address 127.6.98.65:37097
I20240503 13:55:50.678592 8576 tablet_copy_service.cc:140] P a7346c212aa54138970650af7f721d0e: Received BeginTabletCopySession request for tablet 922727f062fc4d6fbb7662a020bc3b65 from peer 45bdf7ba415141cd85008dca119d1cf9 ({username='slave'} at 127.6.98.67:52799)
I20240503 13:55:50.678730 8576 tablet_copy_service.cc:161] P a7346c212aa54138970650af7f721d0e: Beginning new tablet copy session on tablet 922727f062fc4d6fbb7662a020bc3b65 from peer 45bdf7ba415141cd85008dca119d1cf9 at {username='slave'} at 127.6.98.67:52799: session id = 45bdf7ba415141cd85008dca119d1cf9-922727f062fc4d6fbb7662a020bc3b65
I20240503 13:55:50.679823 8576 tablet_copy_source_session.cc:205] T 922727f062fc4d6fbb7662a020bc3b65 P a7346c212aa54138970650af7f721d0e: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:55:50.683065 10378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 922727f062fc4d6fbb7662a020bc3b65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:55:50.688565 10378 tablet_copy_client.cc:783] T 922727f062fc4d6fbb7662a020bc3b65 P 45bdf7ba415141cd85008dca119d1cf9: tablet copy: Starting download of 0 data blocks...
I20240503 13:55:50.688755 10378 tablet_copy_client.cc:647] T 922727f062fc4d6fbb7662a020bc3b65 P 45bdf7ba415141cd85008dca119d1cf9: tablet copy: Starting download of 1 WAL segments...
I20240503 13:55:50.690388 10378 tablet_copy_client.cc:523] T 922727f062fc4d6fbb7662a020bc3b65 P 45bdf7ba415141cd85008dca119d1cf9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:55:50.691951 10378 tablet_bootstrap.cc:492] T 922727f062fc4d6fbb7662a020bc3b65 P 45bdf7ba415141cd85008dca119d1cf9: Bootstrap starting.
I20240503 13:55:50.696903 10378 log.cc:824] T 922727f062fc4d6fbb7662a020bc3b65 P 45bdf7ba415141cd85008dca119d1cf9: Log is configured to *not* fsync() on all Append() calls
I20240503 13:55:50.716303 10378 tablet_bootstrap.cc:492] T 922727f062fc4d6fbb7662a020bc3b65 P 45bdf7ba415141cd85008dca119d1cf9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 13:55:50.716742 10378 tablet_bootstrap.cc:492] T 922727f062fc4d6fbb7662a020bc3b65 P 45bdf7ba415141cd85008dca119d1cf9: Bootstrap complete.
I20240503 13:55:50.717080 10378 ts_tablet_manager.cc:1367] T 922727f062fc4d6fbb7662a020bc3b65 P 45bdf7ba415141cd85008dca119d1cf9: Time spent bootstrapping tablet: real 0.025s user 0.026s sys 0.000s
I20240503 13:55:50.719004 10378 raft_consensus.cc:348] T 922727f062fc4d6fbb7662a020bc3b65 P 45bdf7ba415141cd85008dca119d1cf9 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } }
I20240503 13:55:50.719552 10378 raft_consensus.cc:729] T 922727f062fc4d6fbb7662a020bc3b65 P 45bdf7ba415141cd85008dca119d1cf9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45bdf7ba415141cd85008dca119d1cf9, State: Initialized, Role: FOLLOWER
I20240503 13:55:50.719859 10378 consensus_queue.cc:260] T 922727f062fc4d6fbb7662a020bc3b65 P 45bdf7ba415141cd85008dca119d1cf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } } peers { permanent_uuid: "45bdf7ba415141cd85008dca119d1cf9" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 40171 } }
I20240503 13:55:50.720641 10378 ts_tablet_manager.cc:1398] T 922727f062fc4d6fbb7662a020bc3b65 P 45bdf7ba415141cd85008dca119d1cf9: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240503 13:55:50.721557 8576 tablet_copy_service.cc:342] P a7346c212aa54138970650af7f721d0e: Request end of tablet copy session 45bdf7ba415141cd85008dca119d1cf9-922727f062fc4d6fbb7662a020bc3b65 received from {username='slave'} at 127.6.98.67:52799
I20240503 13:55:50.721647 8576 tablet_copy_service.cc:433] P a7346c212aa54138970650af7f721d0e: ending tablet copy session 45bdf7ba415141cd85008dca119d1cf9-922727f062fc4d6fbb7662a020bc3b65 on tablet 922727f062fc4d6fbb7662a020bc3b65 with peer 45bdf7ba415141cd85008dca119d1cf9
I20240503 13:55:50.766093 10275 heartbeater.cc:503] Master 127.6.98.126:36629 was elected leader, sending a full tablet report...
I20240503 13:55:51.261266 9695 raft_consensus.cc:1203] T 922727f062fc4d6fbb7662a020bc3b65 P 45bdf7ba415141cd85008dca119d1cf9 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:55:51.262586 10389 raft_consensus.cc:2932] T 922727f062fc4d6fbb7662a020bc3b65 P a7346c212aa54138970650af7f721d0e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 45bdf7ba415141cd85008dca119d1cf9 (127.6.98.67) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } } peers { permanent_uuid: "45bdf7ba415141cd85008dca119d1cf9" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 40171 } } }
I20240503 13:55:51.263200 9695 raft_consensus.cc:2932] T 922727f062fc4d6fbb7662a020bc3b65 P 45bdf7ba415141cd85008dca119d1cf9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 45bdf7ba415141cd85008dca119d1cf9 (127.6.98.67) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } } peers { permanent_uuid: "45bdf7ba415141cd85008dca119d1cf9" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 40171 } } }
I20240503 13:55:51.264318 10389 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:35068 (request call id 0) took 1094 ms. Trace:
I20240503 13:55:51.264632 10389 rpcz_store.cc:276] 0503 13:55:50.169801 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 13:55:50.169880 (+ 79us) service_pool.cc:225] Handling call
0503 13:55:50.171239 (+ 1359us) log.cc:842] Serialized 254 byte log entry
0503 13:55:51.264299 (+1093060us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":55,"raft.queue_time_us":110,"raft.run_cpu_time_us":237,"raft.run_wall_time_us":237,"thread_start_us":126,"threads_started":2,"wal-append.queue_time_us":195,"wal-append.run_cpu_time_us":317,"wal-append.run_wall_time_us":1000388}
I20240503 13:55:51.265990 8408 catalog_manager.cc:5484] T 922727f062fc4d6fbb7662a020bc3b65 P 45bdf7ba415141cd85008dca119d1cf9 reported cstate change: config changed from index -1 to 4, VOTER 45bdf7ba415141cd85008dca119d1cf9 (127.6.98.67) added. New cstate: current_term: 1 leader_uuid: "a7346c212aa54138970650af7f721d0e" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } } peers { permanent_uuid: "45bdf7ba415141cd85008dca119d1cf9" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 40171 } } }
I20240503 13:55:51.284523 8687 consensus_queue.cc:237] T f1d7cdc915b3484fbfbc8d4e5ae952b5 P 30acad1886794455a2b5617ba6da9aa1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } } peers { permanent_uuid: "6a0591a0f0b64c418af287c96b5b47b8" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 33575 } }
W20240503 13:55:51.289897 8622 consensus_peers.cc:485] T f1d7cdc915b3484fbfbc8d4e5ae952b5 P 30acad1886794455a2b5617ba6da9aa1 -> Peer 6a0591a0f0b64c418af287c96b5b47b8 (127.6.98.69:33575): Couldn't send request to peer 6a0591a0f0b64c418af287c96b5b47b8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f1d7cdc915b3484fbfbc8d4e5ae952b5. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:55:51.788204 10418 ts_tablet_manager.cc:902] T f1d7cdc915b3484fbfbc8d4e5ae952b5 P 6a0591a0f0b64c418af287c96b5b47b8: Initiating tablet copy from peer 30acad1886794455a2b5617ba6da9aa1 (127.6.98.66:39533)
I20240503 13:55:51.788993 10418 tablet_copy_client.cc:308] T f1d7cdc915b3484fbfbc8d4e5ae952b5 P 6a0591a0f0b64c418af287c96b5b47b8: tablet copy: Beginning tablet copy session from remote peer at address 127.6.98.66:39533
I20240503 13:55:51.792665 8707 tablet_copy_service.cc:140] P 30acad1886794455a2b5617ba6da9aa1: Received BeginTabletCopySession request for tablet f1d7cdc915b3484fbfbc8d4e5ae952b5 from peer 6a0591a0f0b64c418af287c96b5b47b8 ({username='slave'} at 127.6.98.69:38607)
I20240503 13:55:51.792802 8707 tablet_copy_service.cc:161] P 30acad1886794455a2b5617ba6da9aa1: Beginning new tablet copy session on tablet f1d7cdc915b3484fbfbc8d4e5ae952b5 from peer 6a0591a0f0b64c418af287c96b5b47b8 at {username='slave'} at 127.6.98.69:38607: session id = 6a0591a0f0b64c418af287c96b5b47b8-f1d7cdc915b3484fbfbc8d4e5ae952b5
I20240503 13:55:51.793946 8707 tablet_copy_source_session.cc:205] T f1d7cdc915b3484fbfbc8d4e5ae952b5 P 30acad1886794455a2b5617ba6da9aa1: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:55:51.797008 10418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1d7cdc915b3484fbfbc8d4e5ae952b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:55:51.802016 10418 tablet_copy_client.cc:783] T f1d7cdc915b3484fbfbc8d4e5ae952b5 P 6a0591a0f0b64c418af287c96b5b47b8: tablet copy: Starting download of 0 data blocks...
I20240503 13:55:51.802187 10418 tablet_copy_client.cc:647] T f1d7cdc915b3484fbfbc8d4e5ae952b5 P 6a0591a0f0b64c418af287c96b5b47b8: tablet copy: Starting download of 1 WAL segments...
I20240503 13:55:51.803877 10418 tablet_copy_client.cc:523] T f1d7cdc915b3484fbfbc8d4e5ae952b5 P 6a0591a0f0b64c418af287c96b5b47b8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:55:51.805469 10418 tablet_bootstrap.cc:492] T f1d7cdc915b3484fbfbc8d4e5ae952b5 P 6a0591a0f0b64c418af287c96b5b47b8: Bootstrap starting.
I20240503 13:55:51.810496 10418 log.cc:824] T f1d7cdc915b3484fbfbc8d4e5ae952b5 P 6a0591a0f0b64c418af287c96b5b47b8: Log is configured to *not* fsync() on all Append() calls
I20240503 13:55:51.830252 10418 tablet_bootstrap.cc:492] T f1d7cdc915b3484fbfbc8d4e5ae952b5 P 6a0591a0f0b64c418af287c96b5b47b8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 13:55:51.830621 10418 tablet_bootstrap.cc:492] T f1d7cdc915b3484fbfbc8d4e5ae952b5 P 6a0591a0f0b64c418af287c96b5b47b8: Bootstrap complete.
I20240503 13:55:51.830902 10418 ts_tablet_manager.cc:1367] T f1d7cdc915b3484fbfbc8d4e5ae952b5 P 6a0591a0f0b64c418af287c96b5b47b8: Time spent bootstrapping tablet: real 0.026s user 0.021s sys 0.003s
I20240503 13:55:51.832813 10418 raft_consensus.cc:348] T f1d7cdc915b3484fbfbc8d4e5ae952b5 P 6a0591a0f0b64c418af287c96b5b47b8 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } }
I20240503 13:55:51.833338 10418 raft_consensus.cc:729] T f1d7cdc915b3484fbfbc8d4e5ae952b5 P 6a0591a0f0b64c418af287c96b5b47b8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0591a0f0b64c418af287c96b5b47b8, State: Initialized, Role: FOLLOWER
I20240503 13:55:51.833595 10418 consensus_queue.cc:260] T f1d7cdc915b3484fbfbc8d4e5ae952b5 P 6a0591a0f0b64c418af287c96b5b47b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } } peers { permanent_uuid: "6a0591a0f0b64c418af287c96b5b47b8" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 33575 } }
I20240503 13:55:51.834389 10418 ts_tablet_manager.cc:1398] T f1d7cdc915b3484fbfbc8d4e5ae952b5 P 6a0591a0f0b64c418af287c96b5b47b8: Time spent starting tablet: real 0.003s user 0.005s sys 0.000s
I20240503 13:55:51.835320 8707 tablet_copy_service.cc:342] P 30acad1886794455a2b5617ba6da9aa1: Request end of tablet copy session 6a0591a0f0b64c418af287c96b5b47b8-f1d7cdc915b3484fbfbc8d4e5ae952b5 received from {username='slave'} at 127.6.98.69:38607
I20240503 13:55:51.835410 8707 tablet_copy_service.cc:433] P 30acad1886794455a2b5617ba6da9aa1: ending tablet copy session 6a0591a0f0b64c418af287c96b5b47b8-f1d7cdc915b3484fbfbc8d4e5ae952b5 on tablet f1d7cdc915b3484fbfbc8d4e5ae952b5 with peer 6a0591a0f0b64c418af287c96b5b47b8
I20240503 13:55:52.327222 9967 raft_consensus.cc:1203] T f1d7cdc915b3484fbfbc8d4e5ae952b5 P 6a0591a0f0b64c418af287c96b5b47b8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:55:52.328614 10428 raft_consensus.cc:2932] T f1d7cdc915b3484fbfbc8d4e5ae952b5 P 30acad1886794455a2b5617ba6da9aa1 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6a0591a0f0b64c418af287c96b5b47b8 (127.6.98.69) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } } peers { permanent_uuid: "6a0591a0f0b64c418af287c96b5b47b8" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 33575 } } }
I20240503 13:55:52.329080 9967 raft_consensus.cc:2932] T f1d7cdc915b3484fbfbc8d4e5ae952b5 P 6a0591a0f0b64c418af287c96b5b47b8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6a0591a0f0b64c418af287c96b5b47b8 (127.6.98.69) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } } peers { permanent_uuid: "6a0591a0f0b64c418af287c96b5b47b8" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 33575 } } }
I20240503 13:55:52.330030 10428 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:40524 (request call id 0) took 1046 ms. Trace:
I20240503 13:55:52.330155 10428 rpcz_store.cc:276] 0503 13:55:51.283839 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 13:55:51.283896 (+ 57us) service_pool.cc:225] Handling call
0503 13:55:51.285347 (+ 1451us) log.cc:842] Serialized 254 byte log entry
0503 13:55:52.330014 (+1044667us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":53,"raft.queue_time_us":101,"raft.run_cpu_time_us":250,"raft.run_wall_time_us":250,"spinlock_wait_cycles":9088,"thread_start_us":124,"threads_started":2,"wal-append.queue_time_us":160,"wal-append.run_cpu_time_us":312,"wal-append.run_wall_time_us":1000373}
I20240503 13:55:52.331372 8408 catalog_manager.cc:5484] T f1d7cdc915b3484fbfbc8d4e5ae952b5 P 6a0591a0f0b64c418af287c96b5b47b8 reported cstate change: config changed from index -1 to 4, VOTER 6a0591a0f0b64c418af287c96b5b47b8 (127.6.98.69) added. New cstate: current_term: 1 leader_uuid: "30acad1886794455a2b5617ba6da9aa1" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } } peers { permanent_uuid: "6a0591a0f0b64c418af287c96b5b47b8" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 33575 } } }
I20240503 13:55:52.350543 8556 consensus_queue.cc:237] T 30ab7f35240e47ffb4c95fc56dc48fd3 P a7346c212aa54138970650af7f721d0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } } peers { permanent_uuid: "44f093b99a6a403f8e092124ba9fea4c" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 39765 } }
W20240503 13:55:52.356073 8489 consensus_peers.cc:485] T 30ab7f35240e47ffb4c95fc56dc48fd3 P a7346c212aa54138970650af7f721d0e -> Peer 44f093b99a6a403f8e092124ba9fea4c (127.6.98.71:39765): Couldn't send request to peer 44f093b99a6a403f8e092124ba9fea4c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 30ab7f35240e47ffb4c95fc56dc48fd3. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:55:52.963328 10457 ts_tablet_manager.cc:902] T 30ab7f35240e47ffb4c95fc56dc48fd3 P 44f093b99a6a403f8e092124ba9fea4c: Initiating tablet copy from peer a7346c212aa54138970650af7f721d0e (127.6.98.65:37097)
I20240503 13:55:52.964162 10457 tablet_copy_client.cc:308] T 30ab7f35240e47ffb4c95fc56dc48fd3 P 44f093b99a6a403f8e092124ba9fea4c: tablet copy: Beginning tablet copy session from remote peer at address 127.6.98.65:37097
I20240503 13:55:52.968266 8576 tablet_copy_service.cc:140] P a7346c212aa54138970650af7f721d0e: Received BeginTabletCopySession request for tablet 30ab7f35240e47ffb4c95fc56dc48fd3 from peer 44f093b99a6a403f8e092124ba9fea4c ({username='slave'} at 127.6.98.71:33591)
I20240503 13:55:52.968401 8576 tablet_copy_service.cc:161] P a7346c212aa54138970650af7f721d0e: Beginning new tablet copy session on tablet 30ab7f35240e47ffb4c95fc56dc48fd3 from peer 44f093b99a6a403f8e092124ba9fea4c at {username='slave'} at 127.6.98.71:33591: session id = 44f093b99a6a403f8e092124ba9fea4c-30ab7f35240e47ffb4c95fc56dc48fd3
I20240503 13:55:52.969229 8576 tablet_copy_source_session.cc:205] T 30ab7f35240e47ffb4c95fc56dc48fd3 P a7346c212aa54138970650af7f721d0e: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:55:52.972172 10457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30ab7f35240e47ffb4c95fc56dc48fd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:55:52.977815 10457 tablet_copy_client.cc:783] T 30ab7f35240e47ffb4c95fc56dc48fd3 P 44f093b99a6a403f8e092124ba9fea4c: tablet copy: Starting download of 0 data blocks...
I20240503 13:55:52.978008 10457 tablet_copy_client.cc:647] T 30ab7f35240e47ffb4c95fc56dc48fd3 P 44f093b99a6a403f8e092124ba9fea4c: tablet copy: Starting download of 1 WAL segments...
I20240503 13:55:52.979525 10457 tablet_copy_client.cc:523] T 30ab7f35240e47ffb4c95fc56dc48fd3 P 44f093b99a6a403f8e092124ba9fea4c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:55:52.981133 10457 tablet_bootstrap.cc:492] T 30ab7f35240e47ffb4c95fc56dc48fd3 P 44f093b99a6a403f8e092124ba9fea4c: Bootstrap starting.
I20240503 13:55:52.986215 10457 log.cc:824] T 30ab7f35240e47ffb4c95fc56dc48fd3 P 44f093b99a6a403f8e092124ba9fea4c: Log is configured to *not* fsync() on all Append() calls
I20240503 13:55:53.005645 10457 tablet_bootstrap.cc:492] T 30ab7f35240e47ffb4c95fc56dc48fd3 P 44f093b99a6a403f8e092124ba9fea4c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 13:55:53.006042 10457 tablet_bootstrap.cc:492] T 30ab7f35240e47ffb4c95fc56dc48fd3 P 44f093b99a6a403f8e092124ba9fea4c: Bootstrap complete.
I20240503 13:55:53.006359 10457 ts_tablet_manager.cc:1367] T 30ab7f35240e47ffb4c95fc56dc48fd3 P 44f093b99a6a403f8e092124ba9fea4c: Time spent bootstrapping tablet: real 0.025s user 0.024s sys 0.000s
I20240503 13:55:53.008373 10457 raft_consensus.cc:348] T 30ab7f35240e47ffb4c95fc56dc48fd3 P 44f093b99a6a403f8e092124ba9fea4c [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } }
I20240503 13:55:53.008967 10457 raft_consensus.cc:729] T 30ab7f35240e47ffb4c95fc56dc48fd3 P 44f093b99a6a403f8e092124ba9fea4c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44f093b99a6a403f8e092124ba9fea4c, State: Initialized, Role: FOLLOWER
I20240503 13:55:53.009315 10457 consensus_queue.cc:260] T 30ab7f35240e47ffb4c95fc56dc48fd3 P 44f093b99a6a403f8e092124ba9fea4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } } peers { permanent_uuid: "44f093b99a6a403f8e092124ba9fea4c" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 39765 } }
I20240503 13:55:53.010157 10457 ts_tablet_manager.cc:1398] T 30ab7f35240e47ffb4c95fc56dc48fd3 P 44f093b99a6a403f8e092124ba9fea4c: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240503 13:55:53.010802 8576 tablet_copy_service.cc:342] P a7346c212aa54138970650af7f721d0e: Request end of tablet copy session 44f093b99a6a403f8e092124ba9fea4c-30ab7f35240e47ffb4c95fc56dc48fd3 received from {username='slave'} at 127.6.98.71:33591
I20240503 13:55:53.010881 8576 tablet_copy_service.cc:433] P a7346c212aa54138970650af7f721d0e: ending tablet copy session 44f093b99a6a403f8e092124ba9fea4c-30ab7f35240e47ffb4c95fc56dc48fd3 on tablet 30ab7f35240e47ffb4c95fc56dc48fd3 with peer 44f093b99a6a403f8e092124ba9fea4c
I20240503 13:55:53.364768 10229 raft_consensus.cc:1203] T 30ab7f35240e47ffb4c95fc56dc48fd3 P 44f093b99a6a403f8e092124ba9fea4c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:55:53.365983 10466 raft_consensus.cc:2932] T 30ab7f35240e47ffb4c95fc56dc48fd3 P a7346c212aa54138970650af7f721d0e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 44f093b99a6a403f8e092124ba9fea4c (127.6.98.71) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } } peers { permanent_uuid: "44f093b99a6a403f8e092124ba9fea4c" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 39765 } } }
I20240503 13:55:53.366593 10229 raft_consensus.cc:2932] T 30ab7f35240e47ffb4c95fc56dc48fd3 P 44f093b99a6a403f8e092124ba9fea4c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 44f093b99a6a403f8e092124ba9fea4c (127.6.98.71) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } } peers { permanent_uuid: "44f093b99a6a403f8e092124ba9fea4c" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 39765 } } }
I20240503 13:55:53.367564 10466 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:35078 (request call id 0) took 1017 ms. Trace:
I20240503 13:55:53.367630 10466 rpcz_store.cc:276] 0503 13:55:52.350149 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 13:55:52.350208 (+ 59us) service_pool.cc:225] Handling call
0503 13:55:52.351037 (+ 829us) log.cc:842] Serialized 254 byte log entry
0503 13:55:53.367545 (+1016508us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":46,"raft.queue_time_us":43,"raft.run_cpu_time_us":233,"raft.run_wall_time_us":232,"spinlock_wait_cycles":5888,"thread_start_us":76,"threads_started":1,"wal-append.queue_time_us":161,"wal-append.run_cpu_time_us":426,"wal-append.run_wall_time_us":1000536}
I20240503 13:55:53.369815 8412 catalog_manager.cc:5484] T 30ab7f35240e47ffb4c95fc56dc48fd3 P a7346c212aa54138970650af7f721d0e reported cstate change: config changed from index -1 to 4, VOTER 44f093b99a6a403f8e092124ba9fea4c (127.6.98.71) added. New cstate: current_term: 1 leader_uuid: "a7346c212aa54138970650af7f721d0e" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } } peers { permanent_uuid: "44f093b99a6a403f8e092124ba9fea4c" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 39765 } } }
I20240503 13:55:53.386498 8687 consensus_queue.cc:237] T 9864208496e248fdaf0ae116f3dbae6f P 30acad1886794455a2b5617ba6da9aa1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } } peers { permanent_uuid: "a075f746a2e04544931d75c3dc463ef3" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 33391 } }
W20240503 13:55:53.391392 8622 consensus_peers.cc:485] T 9864208496e248fdaf0ae116f3dbae6f P 30acad1886794455a2b5617ba6da9aa1 -> Peer a075f746a2e04544931d75c3dc463ef3 (127.6.98.70:33391): Couldn't send request to peer a075f746a2e04544931d75c3dc463ef3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9864208496e248fdaf0ae116f3dbae6f. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:55:53.975340 10494 ts_tablet_manager.cc:902] T 9864208496e248fdaf0ae116f3dbae6f P a075f746a2e04544931d75c3dc463ef3: Initiating tablet copy from peer 30acad1886794455a2b5617ba6da9aa1 (127.6.98.66:39533)
I20240503 13:55:53.976186 10494 tablet_copy_client.cc:308] T 9864208496e248fdaf0ae116f3dbae6f P a075f746a2e04544931d75c3dc463ef3: tablet copy: Beginning tablet copy session from remote peer at address 127.6.98.66:39533
I20240503 13:55:53.980521 8707 tablet_copy_service.cc:140] P 30acad1886794455a2b5617ba6da9aa1: Received BeginTabletCopySession request for tablet 9864208496e248fdaf0ae116f3dbae6f from peer a075f746a2e04544931d75c3dc463ef3 ({username='slave'} at 127.6.98.70:43227)
I20240503 13:55:53.980659 8707 tablet_copy_service.cc:161] P 30acad1886794455a2b5617ba6da9aa1: Beginning new tablet copy session on tablet 9864208496e248fdaf0ae116f3dbae6f from peer a075f746a2e04544931d75c3dc463ef3 at {username='slave'} at 127.6.98.70:43227: session id = a075f746a2e04544931d75c3dc463ef3-9864208496e248fdaf0ae116f3dbae6f
I20240503 13:55:53.981427 8707 tablet_copy_source_session.cc:205] T 9864208496e248fdaf0ae116f3dbae6f P 30acad1886794455a2b5617ba6da9aa1: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:55:53.984762 10494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9864208496e248fdaf0ae116f3dbae6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:55:53.990218 10494 tablet_copy_client.cc:783] T 9864208496e248fdaf0ae116f3dbae6f P a075f746a2e04544931d75c3dc463ef3: tablet copy: Starting download of 0 data blocks...
I20240503 13:55:53.990410 10494 tablet_copy_client.cc:647] T 9864208496e248fdaf0ae116f3dbae6f P a075f746a2e04544931d75c3dc463ef3: tablet copy: Starting download of 1 WAL segments...
I20240503 13:55:53.991995 10494 tablet_copy_client.cc:523] T 9864208496e248fdaf0ae116f3dbae6f P a075f746a2e04544931d75c3dc463ef3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:55:53.993705 10494 tablet_bootstrap.cc:492] T 9864208496e248fdaf0ae116f3dbae6f P a075f746a2e04544931d75c3dc463ef3: Bootstrap starting.
I20240503 13:55:53.999001 10494 log.cc:824] T 9864208496e248fdaf0ae116f3dbae6f P a075f746a2e04544931d75c3dc463ef3: Log is configured to *not* fsync() on all Append() calls
I20240503 13:55:54.018472 10494 tablet_bootstrap.cc:492] T 9864208496e248fdaf0ae116f3dbae6f P a075f746a2e04544931d75c3dc463ef3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 13:55:54.018899 10494 tablet_bootstrap.cc:492] T 9864208496e248fdaf0ae116f3dbae6f P a075f746a2e04544931d75c3dc463ef3: Bootstrap complete.
I20240503 13:55:54.019204 10494 ts_tablet_manager.cc:1367] T 9864208496e248fdaf0ae116f3dbae6f P a075f746a2e04544931d75c3dc463ef3: Time spent bootstrapping tablet: real 0.026s user 0.023s sys 0.001s
I20240503 13:55:54.021332 10494 raft_consensus.cc:348] T 9864208496e248fdaf0ae116f3dbae6f P a075f746a2e04544931d75c3dc463ef3 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } }
I20240503 13:55:54.021929 10494 raft_consensus.cc:729] T 9864208496e248fdaf0ae116f3dbae6f P a075f746a2e04544931d75c3dc463ef3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a075f746a2e04544931d75c3dc463ef3, State: Initialized, Role: FOLLOWER
I20240503 13:55:54.022236 10494 consensus_queue.cc:260] T 9864208496e248fdaf0ae116f3dbae6f P a075f746a2e04544931d75c3dc463ef3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } } peers { permanent_uuid: "a075f746a2e04544931d75c3dc463ef3" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 33391 } }
I20240503 13:55:54.023066 10494 ts_tablet_manager.cc:1398] T 9864208496e248fdaf0ae116f3dbae6f P a075f746a2e04544931d75c3dc463ef3: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
I20240503 13:55:54.023869 8707 tablet_copy_service.cc:342] P 30acad1886794455a2b5617ba6da9aa1: Request end of tablet copy session a075f746a2e04544931d75c3dc463ef3-9864208496e248fdaf0ae116f3dbae6f received from {username='slave'} at 127.6.98.70:43227
I20240503 13:55:54.023959 8707 tablet_copy_service.cc:433] P 30acad1886794455a2b5617ba6da9aa1: ending tablet copy session a075f746a2e04544931d75c3dc463ef3-9864208496e248fdaf0ae116f3dbae6f on tablet 9864208496e248fdaf0ae116f3dbae6f with peer a075f746a2e04544931d75c3dc463ef3
I20240503 13:55:54.586179 10098 raft_consensus.cc:1203] T 9864208496e248fdaf0ae116f3dbae6f P a075f746a2e04544931d75c3dc463ef3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:55:54.587551 10504 raft_consensus.cc:2932] T 9864208496e248fdaf0ae116f3dbae6f P 30acad1886794455a2b5617ba6da9aa1 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a075f746a2e04544931d75c3dc463ef3 (127.6.98.70) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } } peers { permanent_uuid: "a075f746a2e04544931d75c3dc463ef3" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 33391 } } }
I20240503 13:55:54.588205 10098 raft_consensus.cc:2932] T 9864208496e248fdaf0ae116f3dbae6f P a075f746a2e04544931d75c3dc463ef3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a075f746a2e04544931d75c3dc463ef3 (127.6.98.70) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } } peers { permanent_uuid: "a075f746a2e04544931d75c3dc463ef3" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 33391 } } }
I20240503 13:55:54.588923 10504 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:40538 (request call id 0) took 1202 ms. Trace:
I20240503 13:55:54.588989 10504 rpcz_store.cc:276] 0503 13:55:53.386115 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 13:55:53.386175 (+ 60us) service_pool.cc:225] Handling call
0503 13:55:53.386967 (+ 792us) log.cc:842] Serialized 254 byte log entry
0503 13:55:54.588905 (+1201938us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":46,"raft.queue_time_us":37,"raft.run_cpu_time_us":114,"raft.run_wall_time_us":115,"thread_start_us":71,"threads_started":1,"wal-append.queue_time_us":134,"wal-append.run_cpu_time_us":343,"wal-append.run_wall_time_us":1000414}
I20240503 13:55:54.590243 8412 catalog_manager.cc:5484] T 9864208496e248fdaf0ae116f3dbae6f P 30acad1886794455a2b5617ba6da9aa1 reported cstate change: config changed from index -1 to 4, VOTER a075f746a2e04544931d75c3dc463ef3 (127.6.98.70) added. New cstate: current_term: 1 leader_uuid: "30acad1886794455a2b5617ba6da9aa1" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } } peers { permanent_uuid: "a075f746a2e04544931d75c3dc463ef3" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 33391 } } }
I20240503 13:55:54.614579 8687 consensus_queue.cc:237] T 13c5e1f8f25345cab11a6944eae35f5a P 30acad1886794455a2b5617ba6da9aa1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } } peers { permanent_uuid: "91d1a791a7504f71a4785f6aefe26f1b" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 40653 } }
W20240503 13:55:54.619357 8622 consensus_peers.cc:485] T 13c5e1f8f25345cab11a6944eae35f5a P 30acad1886794455a2b5617ba6da9aa1 -> Peer 91d1a791a7504f71a4785f6aefe26f1b (127.6.98.68:40653): Couldn't send request to peer 91d1a791a7504f71a4785f6aefe26f1b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 13c5e1f8f25345cab11a6944eae35f5a. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:55:55.224920 10531 ts_tablet_manager.cc:902] T 13c5e1f8f25345cab11a6944eae35f5a P 91d1a791a7504f71a4785f6aefe26f1b: Initiating tablet copy from peer 30acad1886794455a2b5617ba6da9aa1 (127.6.98.66:39533)
I20240503 13:55:55.225747 10531 tablet_copy_client.cc:308] T 13c5e1f8f25345cab11a6944eae35f5a P 91d1a791a7504f71a4785f6aefe26f1b: tablet copy: Beginning tablet copy session from remote peer at address 127.6.98.66:39533
I20240503 13:55:55.229748 8707 tablet_copy_service.cc:140] P 30acad1886794455a2b5617ba6da9aa1: Received BeginTabletCopySession request for tablet 13c5e1f8f25345cab11a6944eae35f5a from peer 91d1a791a7504f71a4785f6aefe26f1b ({username='slave'} at 127.6.98.68:55751)
I20240503 13:55:55.229871 8707 tablet_copy_service.cc:161] P 30acad1886794455a2b5617ba6da9aa1: Beginning new tablet copy session on tablet 13c5e1f8f25345cab11a6944eae35f5a from peer 91d1a791a7504f71a4785f6aefe26f1b at {username='slave'} at 127.6.98.68:55751: session id = 91d1a791a7504f71a4785f6aefe26f1b-13c5e1f8f25345cab11a6944eae35f5a
I20240503 13:55:55.230757 8707 tablet_copy_source_session.cc:205] T 13c5e1f8f25345cab11a6944eae35f5a P 30acad1886794455a2b5617ba6da9aa1: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:55:55.232385 10531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13c5e1f8f25345cab11a6944eae35f5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:55:55.234943 10531 tablet_copy_client.cc:783] T 13c5e1f8f25345cab11a6944eae35f5a P 91d1a791a7504f71a4785f6aefe26f1b: tablet copy: Starting download of 0 data blocks...
I20240503 13:55:55.235113 10531 tablet_copy_client.cc:647] T 13c5e1f8f25345cab11a6944eae35f5a P 91d1a791a7504f71a4785f6aefe26f1b: tablet copy: Starting download of 1 WAL segments...
I20240503 13:55:55.236759 10531 tablet_copy_client.cc:523] T 13c5e1f8f25345cab11a6944eae35f5a P 91d1a791a7504f71a4785f6aefe26f1b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:55:55.238005 10531 tablet_bootstrap.cc:492] T 13c5e1f8f25345cab11a6944eae35f5a P 91d1a791a7504f71a4785f6aefe26f1b: Bootstrap starting.
I20240503 13:55:55.259441 10531 tablet_bootstrap.cc:492] T 13c5e1f8f25345cab11a6944eae35f5a P 91d1a791a7504f71a4785f6aefe26f1b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 13:55:55.259810 10531 tablet_bootstrap.cc:492] T 13c5e1f8f25345cab11a6944eae35f5a P 91d1a791a7504f71a4785f6aefe26f1b: Bootstrap complete.
I20240503 13:55:55.259989 10531 ts_tablet_manager.cc:1367] T 13c5e1f8f25345cab11a6944eae35f5a P 91d1a791a7504f71a4785f6aefe26f1b: Time spent bootstrapping tablet: real 0.022s user 0.016s sys 0.004s
I20240503 13:55:55.260541 10531 raft_consensus.cc:348] T 13c5e1f8f25345cab11a6944eae35f5a P 91d1a791a7504f71a4785f6aefe26f1b [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } }
I20240503 13:55:55.260828 10531 raft_consensus.cc:729] T 13c5e1f8f25345cab11a6944eae35f5a P 91d1a791a7504f71a4785f6aefe26f1b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d1a791a7504f71a4785f6aefe26f1b, State: Initialized, Role: FOLLOWER
I20240503 13:55:55.260931 10531 consensus_queue.cc:260] T 13c5e1f8f25345cab11a6944eae35f5a P 91d1a791a7504f71a4785f6aefe26f1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } } peers { permanent_uuid: "91d1a791a7504f71a4785f6aefe26f1b" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 40653 } }
I20240503 13:55:55.261283 10531 ts_tablet_manager.cc:1398] T 13c5e1f8f25345cab11a6944eae35f5a P 91d1a791a7504f71a4785f6aefe26f1b: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 13:55:55.262046 8707 tablet_copy_service.cc:342] P 30acad1886794455a2b5617ba6da9aa1: Request end of tablet copy session 91d1a791a7504f71a4785f6aefe26f1b-13c5e1f8f25345cab11a6944eae35f5a received from {username='slave'} at 127.6.98.68:55751
I20240503 13:55:55.262121 8707 tablet_copy_service.cc:433] P 30acad1886794455a2b5617ba6da9aa1: ending tablet copy session 91d1a791a7504f71a4785f6aefe26f1b-13c5e1f8f25345cab11a6944eae35f5a on tablet 13c5e1f8f25345cab11a6944eae35f5a with peer 91d1a791a7504f71a4785f6aefe26f1b
I20240503 13:55:55.735098 9827 raft_consensus.cc:1203] T 13c5e1f8f25345cab11a6944eae35f5a P 91d1a791a7504f71a4785f6aefe26f1b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:55:55.736378 10540 raft_consensus.cc:2932] T 13c5e1f8f25345cab11a6944eae35f5a P 30acad1886794455a2b5617ba6da9aa1 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 91d1a791a7504f71a4785f6aefe26f1b (127.6.98.68) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } } peers { permanent_uuid: "91d1a791a7504f71a4785f6aefe26f1b" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 40653 } } }
I20240503 13:55:55.736984 9827 raft_consensus.cc:2932] T 13c5e1f8f25345cab11a6944eae35f5a P 91d1a791a7504f71a4785f6aefe26f1b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 91d1a791a7504f71a4785f6aefe26f1b (127.6.98.68) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } } peers { permanent_uuid: "91d1a791a7504f71a4785f6aefe26f1b" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 40653 } } }
I20240503 13:55:55.738041 10540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:40554 (request call id 0) took 1123 ms. Trace:
I20240503 13:55:55.738164 10540 rpcz_store.cc:276] 0503 13:55:54.614190 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 13:55:54.614268 (+ 78us) service_pool.cc:225] Handling call
0503 13:55:54.615055 (+ 787us) log.cc:842] Serialized 254 byte log entry
0503 13:55:55.738024 (+1122969us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":42,"raft.queue_time_us":64,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":126,"thread_start_us":100,"threads_started":1,"wal-append.queue_time_us":173,"wal-append.run_cpu_time_us":346,"wal-append.run_wall_time_us":1000425}
I20240503 13:55:55.740608 8408 catalog_manager.cc:5484] T 13c5e1f8f25345cab11a6944eae35f5a P 91d1a791a7504f71a4785f6aefe26f1b reported cstate change: config changed from index -1 to 4, VOTER 91d1a791a7504f71a4785f6aefe26f1b (127.6.98.68) added. New cstate: current_term: 1 leader_uuid: "30acad1886794455a2b5617ba6da9aa1" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } } peers { permanent_uuid: "91d1a791a7504f71a4785f6aefe26f1b" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 40653 } } }
I20240503 13:55:55.759799 8556 consensus_queue.cc:237] T a9cea7e1421c43fb85282e212e39715d P a7346c212aa54138970650af7f721d0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } } peers { permanent_uuid: "45bdf7ba415141cd85008dca119d1cf9" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 40171 } }
W20240503 13:55:55.761386 8490 consensus_peers.cc:485] T a9cea7e1421c43fb85282e212e39715d P a7346c212aa54138970650af7f721d0e -> Peer 45bdf7ba415141cd85008dca119d1cf9 (127.6.98.67:40171): Couldn't send request to peer 45bdf7ba415141cd85008dca119d1cf9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a9cea7e1421c43fb85282e212e39715d. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:55:56.276051 10565 ts_tablet_manager.cc:902] T a9cea7e1421c43fb85282e212e39715d P 45bdf7ba415141cd85008dca119d1cf9: Initiating tablet copy from peer a7346c212aa54138970650af7f721d0e (127.6.98.65:37097)
I20240503 13:55:56.276680 10565 tablet_copy_client.cc:308] T a9cea7e1421c43fb85282e212e39715d P 45bdf7ba415141cd85008dca119d1cf9: tablet copy: Beginning tablet copy session from remote peer at address 127.6.98.65:37097
I20240503 13:55:56.277386 8576 tablet_copy_service.cc:140] P a7346c212aa54138970650af7f721d0e: Received BeginTabletCopySession request for tablet a9cea7e1421c43fb85282e212e39715d from peer 45bdf7ba415141cd85008dca119d1cf9 ({username='slave'} at 127.6.98.67:52799)
I20240503 13:55:56.277624 8576 tablet_copy_service.cc:161] P a7346c212aa54138970650af7f721d0e: Beginning new tablet copy session on tablet a9cea7e1421c43fb85282e212e39715d from peer 45bdf7ba415141cd85008dca119d1cf9 at {username='slave'} at 127.6.98.67:52799: session id = 45bdf7ba415141cd85008dca119d1cf9-a9cea7e1421c43fb85282e212e39715d
I20240503 13:55:56.278823 8576 tablet_copy_source_session.cc:205] T a9cea7e1421c43fb85282e212e39715d P a7346c212aa54138970650af7f721d0e: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:55:56.279956 10565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9cea7e1421c43fb85282e212e39715d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:55:56.281805 10565 tablet_copy_client.cc:783] T a9cea7e1421c43fb85282e212e39715d P 45bdf7ba415141cd85008dca119d1cf9: tablet copy: Starting download of 0 data blocks...
I20240503 13:55:56.281931 10565 tablet_copy_client.cc:647] T a9cea7e1421c43fb85282e212e39715d P 45bdf7ba415141cd85008dca119d1cf9: tablet copy: Starting download of 1 WAL segments...
I20240503 13:55:56.283257 10565 tablet_copy_client.cc:523] T a9cea7e1421c43fb85282e212e39715d P 45bdf7ba415141cd85008dca119d1cf9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:55:56.284576 10565 tablet_bootstrap.cc:492] T a9cea7e1421c43fb85282e212e39715d P 45bdf7ba415141cd85008dca119d1cf9: Bootstrap starting.
I20240503 13:55:56.301215 10565 tablet_bootstrap.cc:492] T a9cea7e1421c43fb85282e212e39715d P 45bdf7ba415141cd85008dca119d1cf9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 13:55:56.301565 10565 tablet_bootstrap.cc:492] T a9cea7e1421c43fb85282e212e39715d P 45bdf7ba415141cd85008dca119d1cf9: Bootstrap complete.
I20240503 13:55:56.301754 10565 ts_tablet_manager.cc:1367] T a9cea7e1421c43fb85282e212e39715d P 45bdf7ba415141cd85008dca119d1cf9: Time spent bootstrapping tablet: real 0.017s user 0.015s sys 0.001s
I20240503 13:55:56.302327 10565 raft_consensus.cc:348] T a9cea7e1421c43fb85282e212e39715d P 45bdf7ba415141cd85008dca119d1cf9 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } }
I20240503 13:55:56.302568 10565 raft_consensus.cc:729] T a9cea7e1421c43fb85282e212e39715d P 45bdf7ba415141cd85008dca119d1cf9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45bdf7ba415141cd85008dca119d1cf9, State: Initialized, Role: FOLLOWER
I20240503 13:55:56.302681 10565 consensus_queue.cc:260] T a9cea7e1421c43fb85282e212e39715d P 45bdf7ba415141cd85008dca119d1cf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } } peers { permanent_uuid: "45bdf7ba415141cd85008dca119d1cf9" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 40171 } }
I20240503 13:55:56.303153 10565 ts_tablet_manager.cc:1398] T a9cea7e1421c43fb85282e212e39715d P 45bdf7ba415141cd85008dca119d1cf9: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 13:55:56.304047 8576 tablet_copy_service.cc:342] P a7346c212aa54138970650af7f721d0e: Request end of tablet copy session 45bdf7ba415141cd85008dca119d1cf9-a9cea7e1421c43fb85282e212e39715d received from {username='slave'} at 127.6.98.67:52799
I20240503 13:55:56.304172 8576 tablet_copy_service.cc:433] P a7346c212aa54138970650af7f721d0e: ending tablet copy session 45bdf7ba415141cd85008dca119d1cf9-a9cea7e1421c43fb85282e212e39715d on tablet a9cea7e1421c43fb85282e212e39715d with peer 45bdf7ba415141cd85008dca119d1cf9
I20240503 13:55:56.876617 9695 raft_consensus.cc:1203] T a9cea7e1421c43fb85282e212e39715d P 45bdf7ba415141cd85008dca119d1cf9 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:55:56.877730 10573 raft_consensus.cc:2932] T a9cea7e1421c43fb85282e212e39715d P a7346c212aa54138970650af7f721d0e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 45bdf7ba415141cd85008dca119d1cf9 (127.6.98.67) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } } peers { permanent_uuid: "45bdf7ba415141cd85008dca119d1cf9" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 40171 } } }
I20240503 13:55:56.878914 9695 raft_consensus.cc:2932] T a9cea7e1421c43fb85282e212e39715d P 45bdf7ba415141cd85008dca119d1cf9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 45bdf7ba415141cd85008dca119d1cf9 (127.6.98.67) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } } peers { permanent_uuid: "45bdf7ba415141cd85008dca119d1cf9" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 40171 } } }
I20240503 13:55:56.879343 10573 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:35092 (request call id 0) took 1119 ms. Trace:
I20240503 13:55:56.879483 10573 rpcz_store.cc:276] 0503 13:55:55.759448 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 13:55:55.759503 (+ 55us) service_pool.cc:225] Handling call
0503 13:55:55.760237 (+ 734us) log.cc:842] Serialized 254 byte log entry
0503 13:55:56.879314 (+1119077us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":38,"raft.run_cpu_time_us":187,"raft.run_wall_time_us":188,"thread_start_us":79,"threads_started":1,"wal-append.queue_time_us":170,"wal-append.run_cpu_time_us":364,"wal-append.run_wall_time_us":1000446}
I20240503 13:55:56.881184 8408 catalog_manager.cc:5484] T a9cea7e1421c43fb85282e212e39715d P a7346c212aa54138970650af7f721d0e reported cstate change: config changed from index -1 to 4, VOTER 45bdf7ba415141cd85008dca119d1cf9 (127.6.98.67) added. New cstate: current_term: 1 leader_uuid: "a7346c212aa54138970650af7f721d0e" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } } peers { permanent_uuid: "45bdf7ba415141cd85008dca119d1cf9" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 40171 } } }
I20240503 13:55:56.901063 8556 consensus_queue.cc:237] T 8316d131c9ec41e39341f2c1a3c79722 P a7346c212aa54138970650af7f721d0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } } peers { permanent_uuid: "6a0591a0f0b64c418af287c96b5b47b8" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 33575 } }
W20240503 13:55:56.905463 8491 consensus_peers.cc:485] T 8316d131c9ec41e39341f2c1a3c79722 P a7346c212aa54138970650af7f721d0e -> Peer 6a0591a0f0b64c418af287c96b5b47b8 (127.6.98.69:33575): Couldn't send request to peer 6a0591a0f0b64c418af287c96b5b47b8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8316d131c9ec41e39341f2c1a3c79722. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:55:57.416414 10600 ts_tablet_manager.cc:902] T 8316d131c9ec41e39341f2c1a3c79722 P 6a0591a0f0b64c418af287c96b5b47b8: Initiating tablet copy from peer a7346c212aa54138970650af7f721d0e (127.6.98.65:37097)
I20240503 13:55:57.416971 10600 tablet_copy_client.cc:308] T 8316d131c9ec41e39341f2c1a3c79722 P 6a0591a0f0b64c418af287c96b5b47b8: tablet copy: Beginning tablet copy session from remote peer at address 127.6.98.65:37097
I20240503 13:55:57.420477 8576 tablet_copy_service.cc:140] P a7346c212aa54138970650af7f721d0e: Received BeginTabletCopySession request for tablet 8316d131c9ec41e39341f2c1a3c79722 from peer 6a0591a0f0b64c418af287c96b5b47b8 ({username='slave'} at 127.6.98.69:43369)
I20240503 13:55:57.420626 8576 tablet_copy_service.cc:161] P a7346c212aa54138970650af7f721d0e: Beginning new tablet copy session on tablet 8316d131c9ec41e39341f2c1a3c79722 from peer 6a0591a0f0b64c418af287c96b5b47b8 at {username='slave'} at 127.6.98.69:43369: session id = 6a0591a0f0b64c418af287c96b5b47b8-8316d131c9ec41e39341f2c1a3c79722
I20240503 13:55:57.421412 8576 tablet_copy_source_session.cc:205] T 8316d131c9ec41e39341f2c1a3c79722 P a7346c212aa54138970650af7f721d0e: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:55:57.422250 10600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8316d131c9ec41e39341f2c1a3c79722. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:55:57.424023 10600 tablet_copy_client.cc:783] T 8316d131c9ec41e39341f2c1a3c79722 P 6a0591a0f0b64c418af287c96b5b47b8: tablet copy: Starting download of 0 data blocks...
I20240503 13:55:57.424185 10600 tablet_copy_client.cc:647] T 8316d131c9ec41e39341f2c1a3c79722 P 6a0591a0f0b64c418af287c96b5b47b8: tablet copy: Starting download of 1 WAL segments...
I20240503 13:55:57.425338 10600 tablet_copy_client.cc:523] T 8316d131c9ec41e39341f2c1a3c79722 P 6a0591a0f0b64c418af287c96b5b47b8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:55:57.426349 10600 tablet_bootstrap.cc:492] T 8316d131c9ec41e39341f2c1a3c79722 P 6a0591a0f0b64c418af287c96b5b47b8: Bootstrap starting.
I20240503 13:55:57.443925 10600 tablet_bootstrap.cc:492] T 8316d131c9ec41e39341f2c1a3c79722 P 6a0591a0f0b64c418af287c96b5b47b8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 13:55:57.444208 10600 tablet_bootstrap.cc:492] T 8316d131c9ec41e39341f2c1a3c79722 P 6a0591a0f0b64c418af287c96b5b47b8: Bootstrap complete.
I20240503 13:55:57.444391 10600 ts_tablet_manager.cc:1367] T 8316d131c9ec41e39341f2c1a3c79722 P 6a0591a0f0b64c418af287c96b5b47b8: Time spent bootstrapping tablet: real 0.018s user 0.015s sys 0.000s
I20240503 13:55:57.444960 10600 raft_consensus.cc:348] T 8316d131c9ec41e39341f2c1a3c79722 P 6a0591a0f0b64c418af287c96b5b47b8 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } }
I20240503 13:55:57.445156 10600 raft_consensus.cc:729] T 8316d131c9ec41e39341f2c1a3c79722 P 6a0591a0f0b64c418af287c96b5b47b8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0591a0f0b64c418af287c96b5b47b8, State: Initialized, Role: FOLLOWER
I20240503 13:55:57.445235 10600 consensus_queue.cc:260] T 8316d131c9ec41e39341f2c1a3c79722 P 6a0591a0f0b64c418af287c96b5b47b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } } peers { permanent_uuid: "6a0591a0f0b64c418af287c96b5b47b8" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 33575 } }
I20240503 13:55:57.445547 10600 ts_tablet_manager.cc:1398] T 8316d131c9ec41e39341f2c1a3c79722 P 6a0591a0f0b64c418af287c96b5b47b8: Time spent starting tablet: real 0.001s user 0.005s sys 0.000s
I20240503 13:55:57.446163 8576 tablet_copy_service.cc:342] P a7346c212aa54138970650af7f721d0e: Request end of tablet copy session 6a0591a0f0b64c418af287c96b5b47b8-8316d131c9ec41e39341f2c1a3c79722 received from {username='slave'} at 127.6.98.69:43369
I20240503 13:55:57.446255 8576 tablet_copy_service.cc:433] P a7346c212aa54138970650af7f721d0e: ending tablet copy session 6a0591a0f0b64c418af287c96b5b47b8-8316d131c9ec41e39341f2c1a3c79722 on tablet 8316d131c9ec41e39341f2c1a3c79722 with peer 6a0591a0f0b64c418af287c96b5b47b8
I20240503 13:55:57.960074 9967 raft_consensus.cc:1203] T 8316d131c9ec41e39341f2c1a3c79722 P 6a0591a0f0b64c418af287c96b5b47b8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:55:57.961208 10573 raft_consensus.cc:2932] T 8316d131c9ec41e39341f2c1a3c79722 P a7346c212aa54138970650af7f721d0e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6a0591a0f0b64c418af287c96b5b47b8 (127.6.98.69) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } } peers { permanent_uuid: "6a0591a0f0b64c418af287c96b5b47b8" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 33575 } } }
I20240503 13:55:57.961835 9967 raft_consensus.cc:2932] T 8316d131c9ec41e39341f2c1a3c79722 P 6a0591a0f0b64c418af287c96b5b47b8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6a0591a0f0b64c418af287c96b5b47b8 (127.6.98.69) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } } peers { permanent_uuid: "6a0591a0f0b64c418af287c96b5b47b8" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 33575 } } }
I20240503 13:55:57.962606 10573 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:35116 (request call id 0) took 1061 ms. Trace:
I20240503 13:55:57.962719 10573 rpcz_store.cc:276] 0503 13:55:56.900674 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 13:55:56.900733 (+ 59us) service_pool.cc:225] Handling call
0503 13:55:56.901541 (+ 808us) log.cc:842] Serialized 254 byte log entry
0503 13:55:57.962589 (+1061048us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":55,"raft.queue_time_us":39,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":146,"spinlock_wait_cycles":9600,"thread_start_us":98,"threads_started":1,"wal-append.queue_time_us":184,"wal-append.run_cpu_time_us":358,"wal-append.run_wall_time_us":1000438}
I20240503 13:55:57.964359 8412 catalog_manager.cc:5484] T 8316d131c9ec41e39341f2c1a3c79722 P a7346c212aa54138970650af7f721d0e reported cstate change: config changed from index -1 to 4, VOTER 6a0591a0f0b64c418af287c96b5b47b8 (127.6.98.69) added. New cstate: current_term: 1 leader_uuid: "a7346c212aa54138970650af7f721d0e" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } } peers { permanent_uuid: "6a0591a0f0b64c418af287c96b5b47b8" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 33575 } } }
I20240503 13:55:57.981993 8556 consensus_queue.cc:237] T b20e8dd34a25448a88f0658028fad27a P a7346c212aa54138970650af7f721d0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } } peers { permanent_uuid: "44f093b99a6a403f8e092124ba9fea4c" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 39765 } }
W20240503 13:55:57.983346 8489 consensus_peers.cc:485] T b20e8dd34a25448a88f0658028fad27a P a7346c212aa54138970650af7f721d0e -> Peer 44f093b99a6a403f8e092124ba9fea4c (127.6.98.71:39765): Couldn't send request to peer 44f093b99a6a403f8e092124ba9fea4c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b20e8dd34a25448a88f0658028fad27a. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:55:58.569672 10636 ts_tablet_manager.cc:902] T b20e8dd34a25448a88f0658028fad27a P 44f093b99a6a403f8e092124ba9fea4c: Initiating tablet copy from peer a7346c212aa54138970650af7f721d0e (127.6.98.65:37097)
I20240503 13:55:58.570184 10636 tablet_copy_client.cc:308] T b20e8dd34a25448a88f0658028fad27a P 44f093b99a6a403f8e092124ba9fea4c: tablet copy: Beginning tablet copy session from remote peer at address 127.6.98.65:37097
I20240503 13:55:58.570664 8576 tablet_copy_service.cc:140] P a7346c212aa54138970650af7f721d0e: Received BeginTabletCopySession request for tablet b20e8dd34a25448a88f0658028fad27a from peer 44f093b99a6a403f8e092124ba9fea4c ({username='slave'} at 127.6.98.71:33591)
I20240503 13:55:58.570780 8576 tablet_copy_service.cc:161] P a7346c212aa54138970650af7f721d0e: Beginning new tablet copy session on tablet b20e8dd34a25448a88f0658028fad27a from peer 44f093b99a6a403f8e092124ba9fea4c at {username='slave'} at 127.6.98.71:33591: session id = 44f093b99a6a403f8e092124ba9fea4c-b20e8dd34a25448a88f0658028fad27a
I20240503 13:55:58.571909 8576 tablet_copy_source_session.cc:205] T b20e8dd34a25448a88f0658028fad27a P a7346c212aa54138970650af7f721d0e: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:55:58.572913 10636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b20e8dd34a25448a88f0658028fad27a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:55:58.574837 10636 tablet_copy_client.cc:783] T b20e8dd34a25448a88f0658028fad27a P 44f093b99a6a403f8e092124ba9fea4c: tablet copy: Starting download of 0 data blocks...
I20240503 13:55:58.574981 10636 tablet_copy_client.cc:647] T b20e8dd34a25448a88f0658028fad27a P 44f093b99a6a403f8e092124ba9fea4c: tablet copy: Starting download of 1 WAL segments...
I20240503 13:55:58.576148 10636 tablet_copy_client.cc:523] T b20e8dd34a25448a88f0658028fad27a P 44f093b99a6a403f8e092124ba9fea4c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:55:58.577191 10636 tablet_bootstrap.cc:492] T b20e8dd34a25448a88f0658028fad27a P 44f093b99a6a403f8e092124ba9fea4c: Bootstrap starting.
I20240503 13:55:58.595057 10636 tablet_bootstrap.cc:492] T b20e8dd34a25448a88f0658028fad27a P 44f093b99a6a403f8e092124ba9fea4c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 13:55:58.595369 10636 tablet_bootstrap.cc:492] T b20e8dd34a25448a88f0658028fad27a P 44f093b99a6a403f8e092124ba9fea4c: Bootstrap complete.
I20240503 13:55:58.595547 10636 ts_tablet_manager.cc:1367] T b20e8dd34a25448a88f0658028fad27a P 44f093b99a6a403f8e092124ba9fea4c: Time spent bootstrapping tablet: real 0.018s user 0.016s sys 0.000s
I20240503 13:55:58.596077 10636 raft_consensus.cc:348] T b20e8dd34a25448a88f0658028fad27a P 44f093b99a6a403f8e092124ba9fea4c [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } }
I20240503 13:55:58.596297 10636 raft_consensus.cc:729] T b20e8dd34a25448a88f0658028fad27a P 44f093b99a6a403f8e092124ba9fea4c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44f093b99a6a403f8e092124ba9fea4c, State: Initialized, Role: FOLLOWER
I20240503 13:55:58.596395 10636 consensus_queue.cc:260] T b20e8dd34a25448a88f0658028fad27a P 44f093b99a6a403f8e092124ba9fea4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } } peers { permanent_uuid: "44f093b99a6a403f8e092124ba9fea4c" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 39765 } }
I20240503 13:55:58.596765 10636 ts_tablet_manager.cc:1398] T b20e8dd34a25448a88f0658028fad27a P 44f093b99a6a403f8e092124ba9fea4c: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 13:55:58.597424 8576 tablet_copy_service.cc:342] P a7346c212aa54138970650af7f721d0e: Request end of tablet copy session 44f093b99a6a403f8e092124ba9fea4c-b20e8dd34a25448a88f0658028fad27a received from {username='slave'} at 127.6.98.71:33591
I20240503 13:55:58.597508 8576 tablet_copy_service.cc:433] P a7346c212aa54138970650af7f721d0e: ending tablet copy session 44f093b99a6a403f8e092124ba9fea4c-b20e8dd34a25448a88f0658028fad27a on tablet b20e8dd34a25448a88f0658028fad27a with peer 44f093b99a6a403f8e092124ba9fea4c
I20240503 13:55:59.034247 10228 raft_consensus.cc:1203] T b20e8dd34a25448a88f0658028fad27a P 44f093b99a6a403f8e092124ba9fea4c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:55:59.035085 10573 raft_consensus.cc:2932] T b20e8dd34a25448a88f0658028fad27a P a7346c212aa54138970650af7f721d0e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 44f093b99a6a403f8e092124ba9fea4c (127.6.98.71) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } } peers { permanent_uuid: "44f093b99a6a403f8e092124ba9fea4c" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 39765 } } }
I20240503 13:55:59.035595 10228 raft_consensus.cc:2932] T b20e8dd34a25448a88f0658028fad27a P 44f093b99a6a403f8e092124ba9fea4c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 44f093b99a6a403f8e092124ba9fea4c (127.6.98.71) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } } peers { permanent_uuid: "44f093b99a6a403f8e092124ba9fea4c" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 39765 } } }
I20240503 13:55:59.036345 10573 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:39058 (request call id 0) took 1054 ms. Trace:
I20240503 13:55:59.036413 10573 rpcz_store.cc:276] 0503 13:55:57.981596 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 13:55:57.981655 (+ 59us) service_pool.cc:225] Handling call
0503 13:55:57.982376 (+ 721us) log.cc:842] Serialized 254 byte log entry
0503 13:55:59.036330 (+1053954us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":35,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":139,"spinlock_wait_cycles":11136,"thread_start_us":61,"threads_started":1,"wal-append.queue_time_us":138,"wal-append.run_cpu_time_us":333,"wal-append.run_wall_time_us":1000430}
I20240503 13:55:59.038022 8412 catalog_manager.cc:5484] T b20e8dd34a25448a88f0658028fad27a P a7346c212aa54138970650af7f721d0e reported cstate change: config changed from index -1 to 4, VOTER 44f093b99a6a403f8e092124ba9fea4c (127.6.98.71) added. New cstate: current_term: 1 leader_uuid: "a7346c212aa54138970650af7f721d0e" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } } peers { permanent_uuid: "44f093b99a6a403f8e092124ba9fea4c" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 39765 } } }
I20240503 13:55:59.057650 8687 consensus_queue.cc:237] T f8f620ccefff4fa580ad1a1d09c00b59 P 30acad1886794455a2b5617ba6da9aa1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } } peers { permanent_uuid: "a075f746a2e04544931d75c3dc463ef3" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 33391 } }
W20240503 13:55:59.059312 8622 consensus_peers.cc:485] T f8f620ccefff4fa580ad1a1d09c00b59 P 30acad1886794455a2b5617ba6da9aa1 -> Peer a075f746a2e04544931d75c3dc463ef3 (127.6.98.70:33391): Couldn't send request to peer a075f746a2e04544931d75c3dc463ef3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f8f620ccefff4fa580ad1a1d09c00b59. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:55:59.482538 10667 ts_tablet_manager.cc:902] T f8f620ccefff4fa580ad1a1d09c00b59 P a075f746a2e04544931d75c3dc463ef3: Initiating tablet copy from peer 30acad1886794455a2b5617ba6da9aa1 (127.6.98.66:39533)
I20240503 13:55:59.483042 10667 tablet_copy_client.cc:308] T f8f620ccefff4fa580ad1a1d09c00b59 P a075f746a2e04544931d75c3dc463ef3: tablet copy: Beginning tablet copy session from remote peer at address 127.6.98.66:39533
I20240503 13:55:59.483668 8707 tablet_copy_service.cc:140] P 30acad1886794455a2b5617ba6da9aa1: Received BeginTabletCopySession request for tablet f8f620ccefff4fa580ad1a1d09c00b59 from peer a075f746a2e04544931d75c3dc463ef3 ({username='slave'} at 127.6.98.70:43227)
I20240503 13:55:59.483786 8707 tablet_copy_service.cc:161] P 30acad1886794455a2b5617ba6da9aa1: Beginning new tablet copy session on tablet f8f620ccefff4fa580ad1a1d09c00b59 from peer a075f746a2e04544931d75c3dc463ef3 at {username='slave'} at 127.6.98.70:43227: session id = a075f746a2e04544931d75c3dc463ef3-f8f620ccefff4fa580ad1a1d09c00b59
I20240503 13:55:59.484746 8707 tablet_copy_source_session.cc:205] T f8f620ccefff4fa580ad1a1d09c00b59 P 30acad1886794455a2b5617ba6da9aa1: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:55:59.485774 10667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8f620ccefff4fa580ad1a1d09c00b59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:55:59.487661 10667 tablet_copy_client.cc:783] T f8f620ccefff4fa580ad1a1d09c00b59 P a075f746a2e04544931d75c3dc463ef3: tablet copy: Starting download of 0 data blocks...
I20240503 13:55:59.487809 10667 tablet_copy_client.cc:647] T f8f620ccefff4fa580ad1a1d09c00b59 P a075f746a2e04544931d75c3dc463ef3: tablet copy: Starting download of 1 WAL segments...
I20240503 13:55:59.489435 10667 tablet_copy_client.cc:523] T f8f620ccefff4fa580ad1a1d09c00b59 P a075f746a2e04544931d75c3dc463ef3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:55:59.528595 10667 tablet_bootstrap.cc:492] T f8f620ccefff4fa580ad1a1d09c00b59 P a075f746a2e04544931d75c3dc463ef3: Bootstrap starting.
I20240503 13:55:59.548173 10667 tablet_bootstrap.cc:492] T f8f620ccefff4fa580ad1a1d09c00b59 P a075f746a2e04544931d75c3dc463ef3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 13:55:59.548473 10667 tablet_bootstrap.cc:492] T f8f620ccefff4fa580ad1a1d09c00b59 P a075f746a2e04544931d75c3dc463ef3: Bootstrap complete.
I20240503 13:55:59.548684 10667 ts_tablet_manager.cc:1367] T f8f620ccefff4fa580ad1a1d09c00b59 P a075f746a2e04544931d75c3dc463ef3: Time spent bootstrapping tablet: real 0.020s user 0.016s sys 0.004s
I20240503 13:55:59.549288 10667 raft_consensus.cc:348] T f8f620ccefff4fa580ad1a1d09c00b59 P a075f746a2e04544931d75c3dc463ef3 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } }
I20240503 13:55:59.549503 10667 raft_consensus.cc:729] T f8f620ccefff4fa580ad1a1d09c00b59 P a075f746a2e04544931d75c3dc463ef3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a075f746a2e04544931d75c3dc463ef3, State: Initialized, Role: FOLLOWER
I20240503 13:55:59.549605 10667 consensus_queue.cc:260] T f8f620ccefff4fa580ad1a1d09c00b59 P a075f746a2e04544931d75c3dc463ef3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } } peers { permanent_uuid: "a075f746a2e04544931d75c3dc463ef3" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 33391 } }
I20240503 13:55:59.549932 10667 ts_tablet_manager.cc:1398] T f8f620ccefff4fa580ad1a1d09c00b59 P a075f746a2e04544931d75c3dc463ef3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:55:59.550603 8707 tablet_copy_service.cc:342] P 30acad1886794455a2b5617ba6da9aa1: Request end of tablet copy session a075f746a2e04544931d75c3dc463ef3-f8f620ccefff4fa580ad1a1d09c00b59 received from {username='slave'} at 127.6.98.70:43227
I20240503 13:55:59.550689 8707 tablet_copy_service.cc:433] P 30acad1886794455a2b5617ba6da9aa1: ending tablet copy session a075f746a2e04544931d75c3dc463ef3-f8f620ccefff4fa580ad1a1d09c00b59 on tablet f8f620ccefff4fa580ad1a1d09c00b59 with peer a075f746a2e04544931d75c3dc463ef3
I20240503 13:56:00.014904 10098 raft_consensus.cc:1203] T f8f620ccefff4fa580ad1a1d09c00b59 P a075f746a2e04544931d75c3dc463ef3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:56:00.016177 10674 raft_consensus.cc:2932] T f8f620ccefff4fa580ad1a1d09c00b59 P 30acad1886794455a2b5617ba6da9aa1 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a075f746a2e04544931d75c3dc463ef3 (127.6.98.70) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } } peers { permanent_uuid: "a075f746a2e04544931d75c3dc463ef3" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 33391 } } }
I20240503 13:56:00.017117 10098 raft_consensus.cc:2932] T f8f620ccefff4fa580ad1a1d09c00b59 P a075f746a2e04544931d75c3dc463ef3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a075f746a2e04544931d75c3dc463ef3 (127.6.98.70) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } } peers { permanent_uuid: "a075f746a2e04544931d75c3dc463ef3" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 33391 } } }
I20240503 13:56:00.029678 8408 catalog_manager.cc:5484] T f8f620ccefff4fa580ad1a1d09c00b59 P a075f746a2e04544931d75c3dc463ef3 reported cstate change: config changed from index -1 to 4, VOTER a075f746a2e04544931d75c3dc463ef3 (127.6.98.70) added. New cstate: current_term: 1 leader_uuid: "30acad1886794455a2b5617ba6da9aa1" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } } peers { permanent_uuid: "a075f746a2e04544931d75c3dc463ef3" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 33391 } } }
I20240503 13:56:00.098332 9675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:56:00.099143 8666 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:56:00.103853 9947 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:56:00.105278 10078 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:56:00.105669 10209 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:56:00.112468 9807 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:56:00.126140 8536 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:00.157279 8687 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f8f620ccefff4fa580ad1a1d09c00b59"
dest_uuid: "30acad1886794455a2b5617ba6da9aa1"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42932
I20240503 13:56:00.157495 8687 raft_consensus.cc:595] T f8f620ccefff4fa580ad1a1d09c00b59 P 30acad1886794455a2b5617ba6da9aa1 [term 1 LEADER]: Received request to transfer leadership
I20240503 13:56:00.215346 10209 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:56:00.215427 8666 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:56:00.217303 9675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:56:00.232827 9807 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:56:00.234284 10078 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:56:00.238030 8536 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:00.243602 9947 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:56:00.334859 8666 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:56:00.336755 9947 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:56:00.337039 9807 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:56:00.346640 8536 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:00.346890 9675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:56:00.352316 10078 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:56:00.352670 10209 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:56:00.400074 8556 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8316d131c9ec41e39341f2c1a3c79722"
dest_uuid: "a7346c212aa54138970650af7f721d0e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39094
I20240503 13:56:00.400277 8556 raft_consensus.cc:595] T 8316d131c9ec41e39341f2c1a3c79722 P a7346c212aa54138970650af7f721d0e [term 1 LEADER]: Received request to transfer leadership
I20240503 13:56:00.457854 10209 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:56:00.458340 9947 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:56:00.459612 8666 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:56:00.465550 9807 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:56:00.472735 9675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:56:00.482851 8536 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:00.490070 10078 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:56:00.490523 10540 raft_consensus.cc:979] T f8f620ccefff4fa580ad1a1d09c00b59 P 30acad1886794455a2b5617ba6da9aa1: : Instructing follower a075f746a2e04544931d75c3dc463ef3 to start an election
I20240503 13:56:00.490620 10540 raft_consensus.cc:1067] T f8f620ccefff4fa580ad1a1d09c00b59 P 30acad1886794455a2b5617ba6da9aa1 [term 1 LEADER]: Signalling peer a075f746a2e04544931d75c3dc463ef3 to start an election
I20240503 13:56:00.491189 10098 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f8f620ccefff4fa580ad1a1d09c00b59"
dest_uuid: "a075f746a2e04544931d75c3dc463ef3"
from {username='slave'} at 127.6.98.66:48969
I20240503 13:56:00.491398 10098 raft_consensus.cc:482] T f8f620ccefff4fa580ad1a1d09c00b59 P a075f746a2e04544931d75c3dc463ef3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 13:56:00.491518 10098 raft_consensus.cc:3037] T f8f620ccefff4fa580ad1a1d09c00b59 P a075f746a2e04544931d75c3dc463ef3 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:56:00.518596 10573 raft_consensus.cc:979] T 8316d131c9ec41e39341f2c1a3c79722 P a7346c212aa54138970650af7f721d0e: : Instructing follower 6a0591a0f0b64c418af287c96b5b47b8 to start an election
I20240503 13:56:00.518720 10573 raft_consensus.cc:1067] T 8316d131c9ec41e39341f2c1a3c79722 P a7346c212aa54138970650af7f721d0e [term 1 LEADER]: Signalling peer 6a0591a0f0b64c418af287c96b5b47b8 to start an election
I20240503 13:56:00.519331 9967 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8316d131c9ec41e39341f2c1a3c79722"
dest_uuid: "6a0591a0f0b64c418af287c96b5b47b8"
from {username='slave'} at 127.6.98.65:42073
I20240503 13:56:00.519553 9967 raft_consensus.cc:482] T 8316d131c9ec41e39341f2c1a3c79722 P 6a0591a0f0b64c418af287c96b5b47b8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 13:56:00.519639 9967 raft_consensus.cc:3037] T 8316d131c9ec41e39341f2c1a3c79722 P 6a0591a0f0b64c418af287c96b5b47b8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:56:00.536976 10098 raft_consensus.cc:504] T f8f620ccefff4fa580ad1a1d09c00b59 P a075f746a2e04544931d75c3dc463ef3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } } peers { permanent_uuid: "a075f746a2e04544931d75c3dc463ef3" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 33391 } }
I20240503 13:56:00.536967 9967 raft_consensus.cc:504] T 8316d131c9ec41e39341f2c1a3c79722 P 6a0591a0f0b64c418af287c96b5b47b8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } } peers { permanent_uuid: "6a0591a0f0b64c418af287c96b5b47b8" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 33575 } }
I20240503 13:56:00.538148 10098 leader_election.cc:288] T f8f620ccefff4fa580ad1a1d09c00b59 P a075f746a2e04544931d75c3dc463ef3 [CANDIDATE]: Term 2 election: Requested vote from peers 30acad1886794455a2b5617ba6da9aa1 (127.6.98.66:39533)
I20240503 13:56:00.538704 8556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8316d131c9ec41e39341f2c1a3c79722" candidate_uuid: "6a0591a0f0b64c418af287c96b5b47b8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "a7346c212aa54138970650af7f721d0e"
I20240503 13:56:00.538902 8556 raft_consensus.cc:3032] T 8316d131c9ec41e39341f2c1a3c79722 P a7346c212aa54138970650af7f721d0e [term 1 LEADER]: Stepping down as leader of term 1
I20240503 13:56:00.538952 8556 raft_consensus.cc:729] T 8316d131c9ec41e39341f2c1a3c79722 P a7346c212aa54138970650af7f721d0e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a7346c212aa54138970650af7f721d0e, State: Running, Role: LEADER
I20240503 13:56:00.538957 8687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8f620ccefff4fa580ad1a1d09c00b59" candidate_uuid: "a075f746a2e04544931d75c3dc463ef3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "30acad1886794455a2b5617ba6da9aa1"
I20240503 13:56:00.539127 8687 raft_consensus.cc:3032] T f8f620ccefff4fa580ad1a1d09c00b59 P 30acad1886794455a2b5617ba6da9aa1 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 13:56:00.539162 8687 raft_consensus.cc:729] T f8f620ccefff4fa580ad1a1d09c00b59 P 30acad1886794455a2b5617ba6da9aa1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 30acad1886794455a2b5617ba6da9aa1, State: Running, Role: LEADER
I20240503 13:56:00.539136 8556 consensus_queue.cc:260] T 8316d131c9ec41e39341f2c1a3c79722 P a7346c212aa54138970650af7f721d0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } } peers { permanent_uuid: "6a0591a0f0b64c418af287c96b5b47b8" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 33575 } }
I20240503 13:56:00.539276 8556 raft_consensus.cc:3037] T 8316d131c9ec41e39341f2c1a3c79722 P a7346c212aa54138970650af7f721d0e [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:56:00.539336 8687 consensus_queue.cc:260] T f8f620ccefff4fa580ad1a1d09c00b59 P 30acad1886794455a2b5617ba6da9aa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } } peers { permanent_uuid: "a075f746a2e04544931d75c3dc463ef3" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 33391 } }
I20240503 13:56:00.539618 8687 raft_consensus.cc:3037] T f8f620ccefff4fa580ad1a1d09c00b59 P 30acad1886794455a2b5617ba6da9aa1 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:56:00.540400 9967 leader_election.cc:288] T 8316d131c9ec41e39341f2c1a3c79722 P 6a0591a0f0b64c418af287c96b5b47b8 [CANDIDATE]: Term 2 election: Requested vote from peers a7346c212aa54138970650af7f721d0e (127.6.98.65:37097)
I20240503 13:56:00.622686 9947 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:56:00.623442 9675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:56:00.625377 9807 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:56:00.626889 10078 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:56:00.632922 10209 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:56:00.635104 8556 raft_consensus.cc:2454] T 8316d131c9ec41e39341f2c1a3c79722 P a7346c212aa54138970650af7f721d0e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a0591a0f0b64c418af287c96b5b47b8 in term 2.
I20240503 13:56:00.638458 8687 raft_consensus.cc:2454] T f8f620ccefff4fa580ad1a1d09c00b59 P 30acad1886794455a2b5617ba6da9aa1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a075f746a2e04544931d75c3dc463ef3 in term 2.
I20240503 13:56:00.638960 10033 leader_election.cc:302] T f8f620ccefff4fa580ad1a1d09c00b59 P a075f746a2e04544931d75c3dc463ef3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 30acad1886794455a2b5617ba6da9aa1, a075f746a2e04544931d75c3dc463ef3; no voters:
I20240503 13:56:00.639871 8536 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:56:00.641928 9901 leader_election.cc:302] T 8316d131c9ec41e39341f2c1a3c79722 P 6a0591a0f0b64c418af287c96b5b47b8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6a0591a0f0b64c418af287c96b5b47b8, a7346c212aa54138970650af7f721d0e; no voters:
I20240503 13:56:00.642381 10874 raft_consensus.cc:2781] T 8316d131c9ec41e39341f2c1a3c79722 P 6a0591a0f0b64c418af287c96b5b47b8 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:56:00.642625 10874 raft_consensus.cc:686] T 8316d131c9ec41e39341f2c1a3c79722 P 6a0591a0f0b64c418af287c96b5b47b8 [term 2 LEADER]: Becoming Leader. State: Replica: 6a0591a0f0b64c418af287c96b5b47b8, State: Running, Role: LEADER
I20240503 13:56:00.643182 8666 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 13:56:00.643285 10874 consensus_queue.cc:237] T 8316d131c9ec41e39341f2c1a3c79722 P 6a0591a0f0b64c418af287c96b5b47b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } } peers { permanent_uuid: "6a0591a0f0b64c418af287c96b5b47b8" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 33575 } }
I20240503 13:56:00.649724 8406 catalog_manager.cc:5484] T 8316d131c9ec41e39341f2c1a3c79722 P 6a0591a0f0b64c418af287c96b5b47b8 reported cstate change: term changed from 1 to 2, leader changed from a7346c212aa54138970650af7f721d0e (127.6.98.65) to 6a0591a0f0b64c418af287c96b5b47b8 (127.6.98.69). New cstate: current_term: 2 leader_uuid: "6a0591a0f0b64c418af287c96b5b47b8" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } } peers { permanent_uuid: "6a0591a0f0b64c418af287c96b5b47b8" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 33575 } } }
I20240503 13:56:00.661568 10873 raft_consensus.cc:2781] T f8f620ccefff4fa580ad1a1d09c00b59 P a075f746a2e04544931d75c3dc463ef3 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:56:00.661943 10873 raft_consensus.cc:686] T f8f620ccefff4fa580ad1a1d09c00b59 P a075f746a2e04544931d75c3dc463ef3 [term 2 LEADER]: Becoming Leader. State: Replica: a075f746a2e04544931d75c3dc463ef3, State: Running, Role: LEADER
I20240503 13:56:00.662372 10873 consensus_queue.cc:237] T f8f620ccefff4fa580ad1a1d09c00b59 P a075f746a2e04544931d75c3dc463ef3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } } peers { permanent_uuid: "a075f746a2e04544931d75c3dc463ef3" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 33391 } }
I20240503 13:56:00.665647 8411 catalog_manager.cc:5484] T f8f620ccefff4fa580ad1a1d09c00b59 P a075f746a2e04544931d75c3dc463ef3 reported cstate change: term changed from 1 to 2, leader changed from 30acad1886794455a2b5617ba6da9aa1 (127.6.98.66) to a075f746a2e04544931d75c3dc463ef3 (127.6.98.70). New cstate: current_term: 2 leader_uuid: "a075f746a2e04544931d75c3dc463ef3" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } } peers { permanent_uuid: "a075f746a2e04544931d75c3dc463ef3" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 33391 } } }
I20240503 13:56:00.680871 8556 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b20e8dd34a25448a88f0658028fad27a"
dest_uuid: "a7346c212aa54138970650af7f721d0e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39126
I20240503 13:56:00.681049 8556 raft_consensus.cc:595] T b20e8dd34a25448a88f0658028fad27a P a7346c212aa54138970650af7f721d0e [term 1 LEADER]: Received request to transfer leadership
I20240503 13:56:00.733528 9675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:56:00.737318 10209 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:56:00.745425 9947 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:56:00.746237 8536 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:56:00.746646 9807 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:56:00.746999 8666 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 13:56:00.775682 10078 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:56:00.856413 10209 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:56:00.857661 8666 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 13:56:00.860203 9947 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:56:00.862454 9807 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:56:00.873423 8536 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:56:00.874585 9675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:56:00.892654 10078 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:56:00.918105 8687 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f1d7cdc915b3484fbfbc8d4e5ae952b5"
dest_uuid: "30acad1886794455a2b5617ba6da9aa1"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43010
I20240503 13:56:00.918287 8687 raft_consensus.cc:595] T f1d7cdc915b3484fbfbc8d4e5ae952b5 P 30acad1886794455a2b5617ba6da9aa1 [term 1 LEADER]: Received request to transfer leadership
I20240503 13:56:00.974500 10209 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:56:00.977207 8666 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 13:56:00.984692 10078 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:56:00.986238 9807 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:56:00.987458 8536 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:56:00.988456 9947 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:56:01.004210 9675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:56:01.039204 11002 raft_consensus.cc:979] T f1d7cdc915b3484fbfbc8d4e5ae952b5 P 30acad1886794455a2b5617ba6da9aa1: : Instructing follower 6a0591a0f0b64c418af287c96b5b47b8 to start an election
I20240503 13:56:01.039347 11002 raft_consensus.cc:1067] T f1d7cdc915b3484fbfbc8d4e5ae952b5 P 30acad1886794455a2b5617ba6da9aa1 [term 1 LEADER]: Signalling peer 6a0591a0f0b64c418af287c96b5b47b8 to start an election
I20240503 13:56:01.039791 9967 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f1d7cdc915b3484fbfbc8d4e5ae952b5"
dest_uuid: "6a0591a0f0b64c418af287c96b5b47b8"
from {username='slave'} at 127.6.98.66:39819
I20240503 13:56:01.039984 9967 raft_consensus.cc:482] T f1d7cdc915b3484fbfbc8d4e5ae952b5 P 6a0591a0f0b64c418af287c96b5b47b8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 13:56:01.040045 9967 raft_consensus.cc:3037] T f1d7cdc915b3484fbfbc8d4e5ae952b5 P 6a0591a0f0b64c418af287c96b5b47b8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:56:01.041754 9967 raft_consensus.cc:504] T f1d7cdc915b3484fbfbc8d4e5ae952b5 P 6a0591a0f0b64c418af287c96b5b47b8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } } peers { permanent_uuid: "6a0591a0f0b64c418af287c96b5b47b8" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 33575 } }
I20240503 13:56:01.042210 9967 leader_election.cc:288] T f1d7cdc915b3484fbfbc8d4e5ae952b5 P 6a0591a0f0b64c418af287c96b5b47b8 [CANDIDATE]: Term 2 election: Requested vote from peers 30acad1886794455a2b5617ba6da9aa1 (127.6.98.66:39533)
I20240503 13:56:01.042639 8687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d7cdc915b3484fbfbc8d4e5ae952b5" candidate_uuid: "6a0591a0f0b64c418af287c96b5b47b8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "30acad1886794455a2b5617ba6da9aa1"
I20240503 13:56:01.042757 8687 raft_consensus.cc:3032] T f1d7cdc915b3484fbfbc8d4e5ae952b5 P 30acad1886794455a2b5617ba6da9aa1 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 13:56:01.042794 8687 raft_consensus.cc:729] T f1d7cdc915b3484fbfbc8d4e5ae952b5 P 30acad1886794455a2b5617ba6da9aa1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 30acad1886794455a2b5617ba6da9aa1, State: Running, Role: LEADER
I20240503 13:56:01.042939 8687 consensus_queue.cc:260] T f1d7cdc915b3484fbfbc8d4e5ae952b5 P 30acad1886794455a2b5617ba6da9aa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } } peers { permanent_uuid: "6a0591a0f0b64c418af287c96b5b47b8" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 33575 } }
I20240503 13:56:01.043025 8687 raft_consensus.cc:3037] T f1d7cdc915b3484fbfbc8d4e5ae952b5 P 30acad1886794455a2b5617ba6da9aa1 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:56:01.043949 8687 raft_consensus.cc:2454] T f1d7cdc915b3484fbfbc8d4e5ae952b5 P 30acad1886794455a2b5617ba6da9aa1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a0591a0f0b64c418af287c96b5b47b8 in term 2.
I20240503 13:56:01.044284 9902 leader_election.cc:302] T f1d7cdc915b3484fbfbc8d4e5ae952b5 P 6a0591a0f0b64c418af287c96b5b47b8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 30acad1886794455a2b5617ba6da9aa1, 6a0591a0f0b64c418af287c96b5b47b8; no voters:
I20240503 13:56:01.044531 10875 raft_consensus.cc:2781] T f1d7cdc915b3484fbfbc8d4e5ae952b5 P 6a0591a0f0b64c418af287c96b5b47b8 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:56:01.044620 10875 raft_consensus.cc:686] T f1d7cdc915b3484fbfbc8d4e5ae952b5 P 6a0591a0f0b64c418af287c96b5b47b8 [term 2 LEADER]: Becoming Leader. State: Replica: 6a0591a0f0b64c418af287c96b5b47b8, State: Running, Role: LEADER
I20240503 13:56:01.044735 10875 consensus_queue.cc:237] T f1d7cdc915b3484fbfbc8d4e5ae952b5 P 6a0591a0f0b64c418af287c96b5b47b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } } peers { permanent_uuid: "6a0591a0f0b64c418af287c96b5b47b8" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 33575 } }
I20240503 13:56:01.046352 8412 catalog_manager.cc:5484] T f1d7cdc915b3484fbfbc8d4e5ae952b5 P 6a0591a0f0b64c418af287c96b5b47b8 reported cstate change: term changed from 1 to 2, leader changed from 30acad1886794455a2b5617ba6da9aa1 (127.6.98.66) to 6a0591a0f0b64c418af287c96b5b47b8 (127.6.98.69). New cstate: current_term: 2 leader_uuid: "6a0591a0f0b64c418af287c96b5b47b8" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } } peers { permanent_uuid: "6a0591a0f0b64c418af287c96b5b47b8" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 33575 } } }
I20240503 13:56:01.094465 8666 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:56:01.095194 9947 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:01.095932 9675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:56:01.098233 10209 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:56:01.110901 8687 raft_consensus.cc:1261] T f8f620ccefff4fa580ad1a1d09c00b59 P 30acad1886794455a2b5617ba6da9aa1 [term 2 FOLLOWER]: Refusing update from remote peer a075f746a2e04544931d75c3dc463ef3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 13:56:01.111748 10878 consensus_queue.cc:1035] T f8f620ccefff4fa580ad1a1d09c00b59 P a075f746a2e04544931d75c3dc463ef3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:01.115177 10078 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:56:01.116000 9807 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:56:01.124104 8556 raft_consensus.cc:1261] T 8316d131c9ec41e39341f2c1a3c79722 P a7346c212aa54138970650af7f721d0e [term 2 FOLLOWER]: Refusing update from remote peer 6a0591a0f0b64c418af287c96b5b47b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 13:56:01.125150 10874 consensus_queue.cc:1035] T 8316d131c9ec41e39341f2c1a3c79722 P 6a0591a0f0b64c418af287c96b5b47b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:01.126154 8536 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:56:01.136996 10573 raft_consensus.cc:979] T b20e8dd34a25448a88f0658028fad27a P a7346c212aa54138970650af7f721d0e: : Instructing follower 44f093b99a6a403f8e092124ba9fea4c to start an election
I20240503 13:56:01.137086 10573 raft_consensus.cc:1067] T b20e8dd34a25448a88f0658028fad27a P a7346c212aa54138970650af7f721d0e [term 1 LEADER]: Signalling peer 44f093b99a6a403f8e092124ba9fea4c to start an election
I20240503 13:56:01.137658 10228 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b20e8dd34a25448a88f0658028fad27a"
dest_uuid: "44f093b99a6a403f8e092124ba9fea4c"
from {username='slave'} at 127.6.98.65:39991
I20240503 13:56:01.137876 10228 raft_consensus.cc:482] T b20e8dd34a25448a88f0658028fad27a P 44f093b99a6a403f8e092124ba9fea4c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 13:56:01.137969 10228 raft_consensus.cc:3037] T b20e8dd34a25448a88f0658028fad27a P 44f093b99a6a403f8e092124ba9fea4c [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:56:01.141381 10228 raft_consensus.cc:504] T b20e8dd34a25448a88f0658028fad27a P 44f093b99a6a403f8e092124ba9fea4c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } } peers { permanent_uuid: "44f093b99a6a403f8e092124ba9fea4c" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 39765 } }
I20240503 13:56:01.142299 10228 leader_election.cc:288] T b20e8dd34a25448a88f0658028fad27a P 44f093b99a6a403f8e092124ba9fea4c [CANDIDATE]: Term 2 election: Requested vote from peers a7346c212aa54138970650af7f721d0e (127.6.98.65:37097)
I20240503 13:56:01.142652 8556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b20e8dd34a25448a88f0658028fad27a" candidate_uuid: "44f093b99a6a403f8e092124ba9fea4c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "a7346c212aa54138970650af7f721d0e"
I20240503 13:56:01.142788 8556 raft_consensus.cc:3032] T b20e8dd34a25448a88f0658028fad27a P a7346c212aa54138970650af7f721d0e [term 1 LEADER]: Stepping down as leader of term 1
I20240503 13:56:01.142833 8556 raft_consensus.cc:729] T b20e8dd34a25448a88f0658028fad27a P a7346c212aa54138970650af7f721d0e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a7346c212aa54138970650af7f721d0e, State: Running, Role: LEADER
I20240503 13:56:01.142967 8556 consensus_queue.cc:260] T b20e8dd34a25448a88f0658028fad27a P a7346c212aa54138970650af7f721d0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } } peers { permanent_uuid: "44f093b99a6a403f8e092124ba9fea4c" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 39765 } }
I20240503 13:56:01.143098 8556 raft_consensus.cc:3037] T b20e8dd34a25448a88f0658028fad27a P a7346c212aa54138970650af7f721d0e [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:56:01.144021 8556 raft_consensus.cc:2454] T b20e8dd34a25448a88f0658028fad27a P a7346c212aa54138970650af7f721d0e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44f093b99a6a403f8e092124ba9fea4c in term 2.
I20240503 13:56:01.144369 10163 leader_election.cc:302] T b20e8dd34a25448a88f0658028fad27a P 44f093b99a6a403f8e092124ba9fea4c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 44f093b99a6a403f8e092124ba9fea4c, a7346c212aa54138970650af7f721d0e; no voters:
I20240503 13:56:01.144731 11039 raft_consensus.cc:2781] T b20e8dd34a25448a88f0658028fad27a P 44f093b99a6a403f8e092124ba9fea4c [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:56:01.144965 11039 raft_consensus.cc:686] T b20e8dd34a25448a88f0658028fad27a P 44f093b99a6a403f8e092124ba9fea4c [term 2 LEADER]: Becoming Leader. State: Replica: 44f093b99a6a403f8e092124ba9fea4c, State: Running, Role: LEADER
I20240503 13:56:01.145251 11039 consensus_queue.cc:237] T b20e8dd34a25448a88f0658028fad27a P 44f093b99a6a403f8e092124ba9fea4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } } peers { permanent_uuid: "44f093b99a6a403f8e092124ba9fea4c" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 39765 } }
I20240503 13:56:01.151962 8411 catalog_manager.cc:5484] T b20e8dd34a25448a88f0658028fad27a P 44f093b99a6a403f8e092124ba9fea4c reported cstate change: term changed from 1 to 2, leader changed from a7346c212aa54138970650af7f721d0e (127.6.98.65) to 44f093b99a6a403f8e092124ba9fea4c (127.6.98.71). New cstate: current_term: 2 leader_uuid: "44f093b99a6a403f8e092124ba9fea4c" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } } peers { permanent_uuid: "44f093b99a6a403f8e092124ba9fea4c" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 39765 } } }
I20240503 13:56:01.158301 8556 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "922727f062fc4d6fbb7662a020bc3b65"
dest_uuid: "a7346c212aa54138970650af7f721d0e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39178
I20240503 13:56:01.158457 8556 raft_consensus.cc:595] T 922727f062fc4d6fbb7662a020bc3b65 P a7346c212aa54138970650af7f721d0e [term 1 LEADER]: Received request to transfer leadership
I20240503 13:56:01.211544 10209 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:56:01.218418 8666 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:56:01.218461 9675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:56:01.225847 10078 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:56:01.225951 9947 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:01.236346 9807 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:56:01.239634 8536 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 13:56:01.240923 11038 raft_consensus.cc:979] T 922727f062fc4d6fbb7662a020bc3b65 P a7346c212aa54138970650af7f721d0e: : Instructing follower 45bdf7ba415141cd85008dca119d1cf9 to start an election
I20240503 13:56:01.241009 11038 raft_consensus.cc:1067] T 922727f062fc4d6fbb7662a020bc3b65 P a7346c212aa54138970650af7f721d0e [term 1 LEADER]: Signalling peer 45bdf7ba415141cd85008dca119d1cf9 to start an election
I20240503 13:56:01.249161 9695 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "922727f062fc4d6fbb7662a020bc3b65"
dest_uuid: "45bdf7ba415141cd85008dca119d1cf9"
from {username='slave'} at 127.6.98.65:49691
I20240503 13:56:01.249405 9695 raft_consensus.cc:482] T 922727f062fc4d6fbb7662a020bc3b65 P 45bdf7ba415141cd85008dca119d1cf9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 13:56:01.249516 9695 raft_consensus.cc:3037] T 922727f062fc4d6fbb7662a020bc3b65 P 45bdf7ba415141cd85008dca119d1cf9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:56:01.330399 8666 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:56:01.338028 9807 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:56:01.338745 10209 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:56:01.338892 8536 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 13:56:01.339104 9947 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:01.340302 10078 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:56:01.344914 9695 raft_consensus.cc:504] T 922727f062fc4d6fbb7662a020bc3b65 P 45bdf7ba415141cd85008dca119d1cf9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } } peers { permanent_uuid: "45bdf7ba415141cd85008dca119d1cf9" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 40171 } }
I20240503 13:56:01.346072 9695 leader_election.cc:288] T 922727f062fc4d6fbb7662a020bc3b65 P 45bdf7ba415141cd85008dca119d1cf9 [CANDIDATE]: Term 2 election: Requested vote from peers a7346c212aa54138970650af7f721d0e (127.6.98.65:37097)
I20240503 13:56:01.350364 9675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:56:01.361560 8556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "922727f062fc4d6fbb7662a020bc3b65" candidate_uuid: "45bdf7ba415141cd85008dca119d1cf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "a7346c212aa54138970650af7f721d0e"
I20240503 13:56:01.361719 8556 raft_consensus.cc:3032] T 922727f062fc4d6fbb7662a020bc3b65 P a7346c212aa54138970650af7f721d0e [term 1 LEADER]: Stepping down as leader of term 1
I20240503 13:56:01.361769 8556 raft_consensus.cc:729] T 922727f062fc4d6fbb7662a020bc3b65 P a7346c212aa54138970650af7f721d0e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a7346c212aa54138970650af7f721d0e, State: Running, Role: LEADER
I20240503 13:56:01.361927 8556 consensus_queue.cc:260] T 922727f062fc4d6fbb7662a020bc3b65 P a7346c212aa54138970650af7f721d0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } } peers { permanent_uuid: "45bdf7ba415141cd85008dca119d1cf9" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 40171 } }
I20240503 13:56:01.362063 8556 raft_consensus.cc:3037] T 922727f062fc4d6fbb7662a020bc3b65 P a7346c212aa54138970650af7f721d0e [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:56:01.394436 8687 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "13c5e1f8f25345cab11a6944eae35f5a"
dest_uuid: "30acad1886794455a2b5617ba6da9aa1"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43074
I20240503 13:56:01.394604 8687 raft_consensus.cc:595] T 13c5e1f8f25345cab11a6944eae35f5a P 30acad1886794455a2b5617ba6da9aa1 [term 1 LEADER]: Received request to transfer leadership
I20240503 13:56:01.452292 8556 raft_consensus.cc:2454] T 922727f062fc4d6fbb7662a020bc3b65 P a7346c212aa54138970650af7f721d0e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45bdf7ba415141cd85008dca119d1cf9 in term 2.
I20240503 13:56:01.453290 9629 leader_election.cc:302] T 922727f062fc4d6fbb7662a020bc3b65 P 45bdf7ba415141cd85008dca119d1cf9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 45bdf7ba415141cd85008dca119d1cf9, a7346c212aa54138970650af7f721d0e; no voters:
I20240503 13:56:01.455518 9947 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:01.455680 11154 raft_consensus.cc:2781] T 922727f062fc4d6fbb7662a020bc3b65 P 45bdf7ba415141cd85008dca119d1cf9 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:56:01.455796 8666 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:56:01.455839 10209 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:56:01.455906 11154 raft_consensus.cc:686] T 922727f062fc4d6fbb7662a020bc3b65 P 45bdf7ba415141cd85008dca119d1cf9 [term 2 LEADER]: Becoming Leader. State: Replica: 45bdf7ba415141cd85008dca119d1cf9, State: Running, Role: LEADER
I20240503 13:56:01.456290 11154 consensus_queue.cc:237] T 922727f062fc4d6fbb7662a020bc3b65 P 45bdf7ba415141cd85008dca119d1cf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } } peers { permanent_uuid: "45bdf7ba415141cd85008dca119d1cf9" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 40171 } }
I20240503 13:56:01.459218 8412 catalog_manager.cc:5484] T 922727f062fc4d6fbb7662a020bc3b65 P 45bdf7ba415141cd85008dca119d1cf9 reported cstate change: term changed from 1 to 2, leader changed from a7346c212aa54138970650af7f721d0e (127.6.98.65) to 45bdf7ba415141cd85008dca119d1cf9 (127.6.98.67). New cstate: current_term: 2 leader_uuid: "45bdf7ba415141cd85008dca119d1cf9" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } } peers { permanent_uuid: "45bdf7ba415141cd85008dca119d1cf9" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 40171 } } }
I20240503 13:56:01.461609 10078 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:56:01.464016 9675 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:56:01.464838 9807 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:56:01.469066 8536 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:56:01.563899 8556 raft_consensus.cc:1261] T b20e8dd34a25448a88f0658028fad27a P a7346c212aa54138970650af7f721d0e [term 2 FOLLOWER]: Refusing update from remote peer 44f093b99a6a403f8e092124ba9fea4c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 13:56:01.565356 11040 consensus_queue.cc:1035] T b20e8dd34a25448a88f0658028fad27a P 44f093b99a6a403f8e092124ba9fea4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:01.577625 8666 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:56:01.581704 10209 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:56:01.582516 9675 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:56:01.591650 9807 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:56:01.596901 9947 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:01.597429 10078 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:56:01.613837 8536 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:56:01.614933 8687 raft_consensus.cc:1261] T f1d7cdc915b3484fbfbc8d4e5ae952b5 P 30acad1886794455a2b5617ba6da9aa1 [term 2 FOLLOWER]: Refusing update from remote peer 6a0591a0f0b64c418af287c96b5b47b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 13:56:01.615453 10875 consensus_queue.cc:1035] T f1d7cdc915b3484fbfbc8d4e5ae952b5 P 6a0591a0f0b64c418af287c96b5b47b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:01.645877 8556 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "30ab7f35240e47ffb4c95fc56dc48fd3"
dest_uuid: "a7346c212aa54138970650af7f721d0e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39228
I20240503 13:56:01.646056 8556 raft_consensus.cc:595] T 30ab7f35240e47ffb4c95fc56dc48fd3 P a7346c212aa54138970650af7f721d0e [term 1 LEADER]: Received request to transfer leadership
I20240503 13:56:01.700845 9675 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:56:01.703025 8666 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:56:01.704607 10209 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:56:01.709056 9807 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:56:01.710594 10078 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:56:01.722167 8536 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:56:01.730628 9947 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:01.822682 8666 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:56:01.825191 8536 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:56:01.826841 10209 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:56:01.827301 11002 raft_consensus.cc:979] T 13c5e1f8f25345cab11a6944eae35f5a P 30acad1886794455a2b5617ba6da9aa1: : Instructing follower 91d1a791a7504f71a4785f6aefe26f1b to start an election
I20240503 13:56:01.827381 11002 raft_consensus.cc:1067] T 13c5e1f8f25345cab11a6944eae35f5a P 30acad1886794455a2b5617ba6da9aa1 [term 1 LEADER]: Signalling peer 91d1a791a7504f71a4785f6aefe26f1b to start an election
I20240503 13:56:01.827838 9827 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "13c5e1f8f25345cab11a6944eae35f5a"
dest_uuid: "91d1a791a7504f71a4785f6aefe26f1b"
from {username='slave'} at 127.6.98.66:37737
I20240503 13:56:01.828027 9827 raft_consensus.cc:482] T 13c5e1f8f25345cab11a6944eae35f5a P 91d1a791a7504f71a4785f6aefe26f1b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 13:56:01.828076 9827 raft_consensus.cc:3037] T 13c5e1f8f25345cab11a6944eae35f5a P 91d1a791a7504f71a4785f6aefe26f1b [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:56:01.831250 9947 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:01.839577 9675 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:56:01.856415 10078 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:56:01.924095 9827 raft_consensus.cc:504] T 13c5e1f8f25345cab11a6944eae35f5a P 91d1a791a7504f71a4785f6aefe26f1b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } } peers { permanent_uuid: "91d1a791a7504f71a4785f6aefe26f1b" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 40653 } }
I20240503 13:56:01.925185 9827 leader_election.cc:288] T 13c5e1f8f25345cab11a6944eae35f5a P 91d1a791a7504f71a4785f6aefe26f1b [CANDIDATE]: Term 2 election: Requested vote from peers 30acad1886794455a2b5617ba6da9aa1 (127.6.98.66:39533)
I20240503 13:56:01.925628 8687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13c5e1f8f25345cab11a6944eae35f5a" candidate_uuid: "91d1a791a7504f71a4785f6aefe26f1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "30acad1886794455a2b5617ba6da9aa1"
I20240503 13:56:01.925766 8687 raft_consensus.cc:3032] T 13c5e1f8f25345cab11a6944eae35f5a P 30acad1886794455a2b5617ba6da9aa1 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 13:56:01.925815 8687 raft_consensus.cc:729] T 13c5e1f8f25345cab11a6944eae35f5a P 30acad1886794455a2b5617ba6da9aa1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 30acad1886794455a2b5617ba6da9aa1, State: Running, Role: LEADER
I20240503 13:56:01.925992 8687 consensus_queue.cc:260] T 13c5e1f8f25345cab11a6944eae35f5a P 30acad1886794455a2b5617ba6da9aa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } } peers { permanent_uuid: "91d1a791a7504f71a4785f6aefe26f1b" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 40653 } }
I20240503 13:56:01.926154 8687 raft_consensus.cc:3037] T 13c5e1f8f25345cab11a6944eae35f5a P 30acad1886794455a2b5617ba6da9aa1 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:56:01.926581 9807 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:56:01.927284 8687 raft_consensus.cc:2454] T 13c5e1f8f25345cab11a6944eae35f5a P 30acad1886794455a2b5617ba6da9aa1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d1a791a7504f71a4785f6aefe26f1b in term 2.
I20240503 13:56:01.927692 9761 leader_election.cc:302] T 13c5e1f8f25345cab11a6944eae35f5a P 91d1a791a7504f71a4785f6aefe26f1b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 30acad1886794455a2b5617ba6da9aa1, 91d1a791a7504f71a4785f6aefe26f1b; no voters:
I20240503 13:56:01.928023 11273 raft_consensus.cc:2781] T 13c5e1f8f25345cab11a6944eae35f5a P 91d1a791a7504f71a4785f6aefe26f1b [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:56:01.928184 11273 raft_consensus.cc:686] T 13c5e1f8f25345cab11a6944eae35f5a P 91d1a791a7504f71a4785f6aefe26f1b [term 2 LEADER]: Becoming Leader. State: Replica: 91d1a791a7504f71a4785f6aefe26f1b, State: Running, Role: LEADER
I20240503 13:56:01.928447 11273 consensus_queue.cc:237] T 13c5e1f8f25345cab11a6944eae35f5a P 91d1a791a7504f71a4785f6aefe26f1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } } peers { permanent_uuid: "91d1a791a7504f71a4785f6aefe26f1b" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 40653 } }
I20240503 13:56:01.930333 8411 catalog_manager.cc:5484] T 13c5e1f8f25345cab11a6944eae35f5a P 91d1a791a7504f71a4785f6aefe26f1b reported cstate change: term changed from 1 to 2, leader changed from 30acad1886794455a2b5617ba6da9aa1 (127.6.98.66) to 91d1a791a7504f71a4785f6aefe26f1b (127.6.98.68). New cstate: current_term: 2 leader_uuid: "91d1a791a7504f71a4785f6aefe26f1b" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } } peers { permanent_uuid: "91d1a791a7504f71a4785f6aefe26f1b" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 40653 } } }
I20240503 13:56:01.948555 8687 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9864208496e248fdaf0ae116f3dbae6f"
dest_uuid: "30acad1886794455a2b5617ba6da9aa1"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43128
I20240503 13:56:01.948715 8687 raft_consensus.cc:595] T 9864208496e248fdaf0ae116f3dbae6f P 30acad1886794455a2b5617ba6da9aa1 [term 1 LEADER]: Received request to transfer leadership
I20240503 13:56:02.002380 8666 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:56:02.004578 9947 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:02.008914 9675 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:56:02.010483 10209 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:56:02.019891 10078 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:56:02.029085 9807 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:02.038470 8536 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:56:02.052736 8556 raft_consensus.cc:1261] T 922727f062fc4d6fbb7662a020bc3b65 P a7346c212aa54138970650af7f721d0e [term 2 FOLLOWER]: Refusing update from remote peer 45bdf7ba415141cd85008dca119d1cf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 13:56:02.053388 11313 consensus_queue.cc:1035] T 922727f062fc4d6fbb7662a020bc3b65 P 45bdf7ba415141cd85008dca119d1cf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:02.094851 11334 raft_consensus.cc:979] T 30ab7f35240e47ffb4c95fc56dc48fd3 P a7346c212aa54138970650af7f721d0e: : Instructing follower 44f093b99a6a403f8e092124ba9fea4c to start an election
I20240503 13:56:02.094962 11334 raft_consensus.cc:1067] T 30ab7f35240e47ffb4c95fc56dc48fd3 P a7346c212aa54138970650af7f721d0e [term 1 LEADER]: Signalling peer 44f093b99a6a403f8e092124ba9fea4c to start an election
I20240503 13:56:02.095407 10228 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "30ab7f35240e47ffb4c95fc56dc48fd3"
dest_uuid: "44f093b99a6a403f8e092124ba9fea4c"
from {username='slave'} at 127.6.98.65:39991
I20240503 13:56:02.095556 10228 raft_consensus.cc:482] T 30ab7f35240e47ffb4c95fc56dc48fd3 P 44f093b99a6a403f8e092124ba9fea4c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 13:56:02.095628 10228 raft_consensus.cc:3037] T 30ab7f35240e47ffb4c95fc56dc48fd3 P 44f093b99a6a403f8e092124ba9fea4c [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:56:02.131310 10228 raft_consensus.cc:504] T 30ab7f35240e47ffb4c95fc56dc48fd3 P 44f093b99a6a403f8e092124ba9fea4c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } } peers { permanent_uuid: "44f093b99a6a403f8e092124ba9fea4c" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 39765 } }
I20240503 13:56:02.131711 10228 leader_election.cc:288] T 30ab7f35240e47ffb4c95fc56dc48fd3 P 44f093b99a6a403f8e092124ba9fea4c [CANDIDATE]: Term 2 election: Requested vote from peers a7346c212aa54138970650af7f721d0e (127.6.98.65:37097)
I20240503 13:56:02.132655 10078 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:56:02.133045 8556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30ab7f35240e47ffb4c95fc56dc48fd3" candidate_uuid: "44f093b99a6a403f8e092124ba9fea4c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "a7346c212aa54138970650af7f721d0e"
I20240503 13:56:02.133186 8556 raft_consensus.cc:3032] T 30ab7f35240e47ffb4c95fc56dc48fd3 P a7346c212aa54138970650af7f721d0e [term 1 LEADER]: Stepping down as leader of term 1
I20240503 13:56:02.133231 8556 raft_consensus.cc:729] T 30ab7f35240e47ffb4c95fc56dc48fd3 P a7346c212aa54138970650af7f721d0e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a7346c212aa54138970650af7f721d0e, State: Running, Role: LEADER
I20240503 13:56:02.133380 8556 consensus_queue.cc:260] T 30ab7f35240e47ffb4c95fc56dc48fd3 P a7346c212aa54138970650af7f721d0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } } peers { permanent_uuid: "44f093b99a6a403f8e092124ba9fea4c" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 39765 } }
I20240503 13:56:02.133505 8556 raft_consensus.cc:3037] T 30ab7f35240e47ffb4c95fc56dc48fd3 P a7346c212aa54138970650af7f721d0e [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:56:02.133504 9675 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:56:02.135710 9807 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:02.136852 10209 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:56:02.151837 8666 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:56:02.151875 9947 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:02.227875 8556 raft_consensus.cc:2454] T 30ab7f35240e47ffb4c95fc56dc48fd3 P a7346c212aa54138970650af7f721d0e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44f093b99a6a403f8e092124ba9fea4c in term 2.
I20240503 13:56:02.228454 10163 leader_election.cc:302] T 30ab7f35240e47ffb4c95fc56dc48fd3 P 44f093b99a6a403f8e092124ba9fea4c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 44f093b99a6a403f8e092124ba9fea4c, a7346c212aa54138970650af7f721d0e; no voters:
I20240503 13:56:02.228801 11039 raft_consensus.cc:2781] T 30ab7f35240e47ffb4c95fc56dc48fd3 P 44f093b99a6a403f8e092124ba9fea4c [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:56:02.228982 11039 raft_consensus.cc:686] T 30ab7f35240e47ffb4c95fc56dc48fd3 P 44f093b99a6a403f8e092124ba9fea4c [term 2 LEADER]: Becoming Leader. State: Replica: 44f093b99a6a403f8e092124ba9fea4c, State: Running, Role: LEADER
I20240503 13:56:02.229135 11039 consensus_queue.cc:237] T 30ab7f35240e47ffb4c95fc56dc48fd3 P 44f093b99a6a403f8e092124ba9fea4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } } peers { permanent_uuid: "44f093b99a6a403f8e092124ba9fea4c" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 39765 } }
I20240503 13:56:02.230065 8536 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:56:02.230847 8410 catalog_manager.cc:5484] T 30ab7f35240e47ffb4c95fc56dc48fd3 P 44f093b99a6a403f8e092124ba9fea4c reported cstate change: term changed from 1 to 2, leader changed from a7346c212aa54138970650af7f721d0e (127.6.98.65) to 44f093b99a6a403f8e092124ba9fea4c (127.6.98.71). New cstate: current_term: 2 leader_uuid: "44f093b99a6a403f8e092124ba9fea4c" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } } peers { permanent_uuid: "44f093b99a6a403f8e092124ba9fea4c" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 39765 } } }
I20240503 13:56:02.251751 8556 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a9cea7e1421c43fb85282e212e39715d"
dest_uuid: "a7346c212aa54138970650af7f721d0e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39266
I20240503 13:56:02.251931 8556 raft_consensus.cc:595] T a9cea7e1421c43fb85282e212e39715d P a7346c212aa54138970650af7f721d0e [term 1 LEADER]: Received request to transfer leadership
I20240503 13:56:02.304505 10209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:02.308238 9675 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:56:02.309077 8687 raft_consensus.cc:1261] T 13c5e1f8f25345cab11a6944eae35f5a P 30acad1886794455a2b5617ba6da9aa1 [term 2 FOLLOWER]: Refusing update from remote peer 91d1a791a7504f71a4785f6aefe26f1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 13:56:02.309686 11273 consensus_queue.cc:1035] T 13c5e1f8f25345cab11a6944eae35f5a P 91d1a791a7504f71a4785f6aefe26f1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:02.310410 11002 raft_consensus.cc:979] T 9864208496e248fdaf0ae116f3dbae6f P 30acad1886794455a2b5617ba6da9aa1: : Instructing follower a075f746a2e04544931d75c3dc463ef3 to start an election
I20240503 13:56:02.310492 11002 raft_consensus.cc:1067] T 9864208496e248fdaf0ae116f3dbae6f P 30acad1886794455a2b5617ba6da9aa1 [term 1 LEADER]: Signalling peer a075f746a2e04544931d75c3dc463ef3 to start an election
I20240503 13:56:02.310653 9947 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:02.311033 8666 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:56:02.313033 9807 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:02.313365 10098 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9864208496e248fdaf0ae116f3dbae6f"
dest_uuid: "a075f746a2e04544931d75c3dc463ef3"
from {username='slave'} at 127.6.98.66:48969
I20240503 13:56:02.313498 10098 raft_consensus.cc:482] T 9864208496e248fdaf0ae116f3dbae6f P a075f746a2e04544931d75c3dc463ef3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 13:56:02.313555 10098 raft_consensus.cc:3037] T 9864208496e248fdaf0ae116f3dbae6f P a075f746a2e04544931d75c3dc463ef3 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:56:02.321440 10078 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:56:02.330488 8536 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:56:02.331940 10098 raft_consensus.cc:504] T 9864208496e248fdaf0ae116f3dbae6f P a075f746a2e04544931d75c3dc463ef3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } } peers { permanent_uuid: "a075f746a2e04544931d75c3dc463ef3" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 33391 } }
I20240503 13:56:02.332336 10098 leader_election.cc:288] T 9864208496e248fdaf0ae116f3dbae6f P a075f746a2e04544931d75c3dc463ef3 [CANDIDATE]: Term 2 election: Requested vote from peers 30acad1886794455a2b5617ba6da9aa1 (127.6.98.66:39533)
I20240503 13:56:02.332996 8687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9864208496e248fdaf0ae116f3dbae6f" candidate_uuid: "a075f746a2e04544931d75c3dc463ef3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "30acad1886794455a2b5617ba6da9aa1"
I20240503 13:56:02.333132 8687 raft_consensus.cc:3032] T 9864208496e248fdaf0ae116f3dbae6f P 30acad1886794455a2b5617ba6da9aa1 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 13:56:02.333165 8687 raft_consensus.cc:729] T 9864208496e248fdaf0ae116f3dbae6f P 30acad1886794455a2b5617ba6da9aa1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 30acad1886794455a2b5617ba6da9aa1, State: Running, Role: LEADER
I20240503 13:56:02.333304 8687 consensus_queue.cc:260] T 9864208496e248fdaf0ae116f3dbae6f P 30acad1886794455a2b5617ba6da9aa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } } peers { permanent_uuid: "a075f746a2e04544931d75c3dc463ef3" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 33391 } }
I20240503 13:56:02.333428 8687 raft_consensus.cc:3037] T 9864208496e248fdaf0ae116f3dbae6f P 30acad1886794455a2b5617ba6da9aa1 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:56:02.334431 8687 raft_consensus.cc:2454] T 9864208496e248fdaf0ae116f3dbae6f P 30acad1886794455a2b5617ba6da9aa1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a075f746a2e04544931d75c3dc463ef3 in term 2.
I20240503 13:56:02.334951 10033 leader_election.cc:302] T 9864208496e248fdaf0ae116f3dbae6f P a075f746a2e04544931d75c3dc463ef3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 30acad1886794455a2b5617ba6da9aa1, a075f746a2e04544931d75c3dc463ef3; no voters:
I20240503 13:56:02.338868 11399 raft_consensus.cc:2781] T 9864208496e248fdaf0ae116f3dbae6f P a075f746a2e04544931d75c3dc463ef3 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:56:02.339098 11399 raft_consensus.cc:686] T 9864208496e248fdaf0ae116f3dbae6f P a075f746a2e04544931d75c3dc463ef3 [term 2 LEADER]: Becoming Leader. State: Replica: a075f746a2e04544931d75c3dc463ef3, State: Running, Role: LEADER
I20240503 13:56:02.339295 11399 consensus_queue.cc:237] T 9864208496e248fdaf0ae116f3dbae6f P a075f746a2e04544931d75c3dc463ef3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } } peers { permanent_uuid: "a075f746a2e04544931d75c3dc463ef3" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 33391 } }
I20240503 13:56:02.342272 8412 catalog_manager.cc:5484] T 9864208496e248fdaf0ae116f3dbae6f P a075f746a2e04544931d75c3dc463ef3 reported cstate change: term changed from 1 to 2, leader changed from 30acad1886794455a2b5617ba6da9aa1 (127.6.98.66) to a075f746a2e04544931d75c3dc463ef3 (127.6.98.70). New cstate: current_term: 2 leader_uuid: "a075f746a2e04544931d75c3dc463ef3" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } } peers { permanent_uuid: "a075f746a2e04544931d75c3dc463ef3" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 33391 } } }
I20240503 13:56:02.377993 11038 raft_consensus.cc:979] T a9cea7e1421c43fb85282e212e39715d P a7346c212aa54138970650af7f721d0e: : Instructing follower 45bdf7ba415141cd85008dca119d1cf9 to start an election
I20240503 13:56:02.378108 11038 raft_consensus.cc:1067] T a9cea7e1421c43fb85282e212e39715d P a7346c212aa54138970650af7f721d0e [term 1 LEADER]: Signalling peer 45bdf7ba415141cd85008dca119d1cf9 to start an election
I20240503 13:56:02.378531 9695 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a9cea7e1421c43fb85282e212e39715d"
dest_uuid: "45bdf7ba415141cd85008dca119d1cf9"
from {username='slave'} at 127.6.98.65:49691
I20240503 13:56:02.378664 9695 raft_consensus.cc:482] T a9cea7e1421c43fb85282e212e39715d P 45bdf7ba415141cd85008dca119d1cf9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 13:56:02.378708 9695 raft_consensus.cc:3037] T a9cea7e1421c43fb85282e212e39715d P 45bdf7ba415141cd85008dca119d1cf9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:56:02.431133 9695 raft_consensus.cc:504] T a9cea7e1421c43fb85282e212e39715d P 45bdf7ba415141cd85008dca119d1cf9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } } peers { permanent_uuid: "45bdf7ba415141cd85008dca119d1cf9" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 40171 } }
I20240503 13:56:02.431576 9695 leader_election.cc:288] T a9cea7e1421c43fb85282e212e39715d P 45bdf7ba415141cd85008dca119d1cf9 [CANDIDATE]: Term 2 election: Requested vote from peers a7346c212aa54138970650af7f721d0e (127.6.98.65:37097)
I20240503 13:56:02.432087 8556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9cea7e1421c43fb85282e212e39715d" candidate_uuid: "45bdf7ba415141cd85008dca119d1cf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "a7346c212aa54138970650af7f721d0e"
I20240503 13:56:02.432211 8556 raft_consensus.cc:3032] T a9cea7e1421c43fb85282e212e39715d P a7346c212aa54138970650af7f721d0e [term 1 LEADER]: Stepping down as leader of term 1
I20240503 13:56:02.432241 8556 raft_consensus.cc:729] T a9cea7e1421c43fb85282e212e39715d P a7346c212aa54138970650af7f721d0e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a7346c212aa54138970650af7f721d0e, State: Running, Role: LEADER
I20240503 13:56:02.432404 8556 consensus_queue.cc:260] T a9cea7e1421c43fb85282e212e39715d P a7346c212aa54138970650af7f721d0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } } peers { permanent_uuid: "45bdf7ba415141cd85008dca119d1cf9" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 40171 } }
I20240503 13:56:02.432587 8556 raft_consensus.cc:3037] T a9cea7e1421c43fb85282e212e39715d P a7346c212aa54138970650af7f721d0e [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:56:02.433377 8556 raft_consensus.cc:2454] T a9cea7e1421c43fb85282e212e39715d P a7346c212aa54138970650af7f721d0e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45bdf7ba415141cd85008dca119d1cf9 in term 2.
I20240503 13:56:02.433815 9629 leader_election.cc:302] T a9cea7e1421c43fb85282e212e39715d P 45bdf7ba415141cd85008dca119d1cf9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 45bdf7ba415141cd85008dca119d1cf9, a7346c212aa54138970650af7f721d0e; no voters:
I20240503 13:56:02.434088 11315 raft_consensus.cc:2781] T a9cea7e1421c43fb85282e212e39715d P 45bdf7ba415141cd85008dca119d1cf9 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:56:02.434192 11315 raft_consensus.cc:686] T a9cea7e1421c43fb85282e212e39715d P 45bdf7ba415141cd85008dca119d1cf9 [term 2 LEADER]: Becoming Leader. State: Replica: 45bdf7ba415141cd85008dca119d1cf9, State: Running, Role: LEADER
I20240503 13:56:02.434329 11315 consensus_queue.cc:237] T a9cea7e1421c43fb85282e212e39715d P 45bdf7ba415141cd85008dca119d1cf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } } peers { permanent_uuid: "45bdf7ba415141cd85008dca119d1cf9" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 40171 } }
I20240503 13:56:02.435952 8412 catalog_manager.cc:5484] T a9cea7e1421c43fb85282e212e39715d P 45bdf7ba415141cd85008dca119d1cf9 reported cstate change: term changed from 1 to 2, leader changed from a7346c212aa54138970650af7f721d0e (127.6.98.65) to 45bdf7ba415141cd85008dca119d1cf9 (127.6.98.67). New cstate: current_term: 2 leader_uuid: "45bdf7ba415141cd85008dca119d1cf9" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } } peers { permanent_uuid: "45bdf7ba415141cd85008dca119d1cf9" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 40171 } } }
I20240503 13:56:02.614807 8556 raft_consensus.cc:1261] T 30ab7f35240e47ffb4c95fc56dc48fd3 P a7346c212aa54138970650af7f721d0e [term 2 FOLLOWER]: Refusing update from remote peer 44f093b99a6a403f8e092124ba9fea4c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 13:56:02.615607 11039 consensus_queue.cc:1035] T 30ab7f35240e47ffb4c95fc56dc48fd3 P 44f093b99a6a403f8e092124ba9fea4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:02.637526 8666 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:56:02.640697 9675 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:02.640736 10209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:02.642557 9807 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:02.644783 8536 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:56:02.649041 9947 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:02.652948 10078 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:02.702317 10097 consensus_queue.cc:237] T f8f620ccefff4fa580ad1a1d09c00b59 P a075f746a2e04544931d75c3dc463ef3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a075f746a2e04544931d75c3dc463ef3" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 33391 } }
I20240503 13:56:02.703208 11400 raft_consensus.cc:2932] T f8f620ccefff4fa580ad1a1d09c00b59 P a075f746a2e04544931d75c3dc463ef3 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 30acad1886794455a2b5617ba6da9aa1 (127.6.98.66) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a075f746a2e04544931d75c3dc463ef3" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 33391 } } }
I20240503 13:56:02.719894 8687 raft_consensus.cc:1261] T 9864208496e248fdaf0ae116f3dbae6f P 30acad1886794455a2b5617ba6da9aa1 [term 2 FOLLOWER]: Refusing update from remote peer a075f746a2e04544931d75c3dc463ef3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 13:56:02.720360 11398 consensus_queue.cc:1035] T 9864208496e248fdaf0ae116f3dbae6f P a075f746a2e04544931d75c3dc463ef3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:02.733749 8411 catalog_manager.cc:5484] T f8f620ccefff4fa580ad1a1d09c00b59 P a075f746a2e04544931d75c3dc463ef3 reported cstate change: config changed from index 4 to 6, VOTER 30acad1886794455a2b5617ba6da9aa1 (127.6.98.66) evicted. New cstate: current_term: 2 leader_uuid: "a075f746a2e04544931d75c3dc463ef3" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a075f746a2e04544931d75c3dc463ef3" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 33391 } } }
I20240503 13:56:02.736838 8666 tablet_service.cc:1514] Processing DeleteTablet for tablet f8f620ccefff4fa580ad1a1d09c00b59 with delete_type TABLET_DATA_TOMBSTONED (TS 30acad1886794455a2b5617ba6da9aa1 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:53700
I20240503 13:56:02.737637 11466 tablet_replica.cc:330] T f8f620ccefff4fa580ad1a1d09c00b59 P 30acad1886794455a2b5617ba6da9aa1: stopping tablet replica
I20240503 13:56:02.737978 11466 raft_consensus.cc:2229] T f8f620ccefff4fa580ad1a1d09c00b59 P 30acad1886794455a2b5617ba6da9aa1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:02.738158 11466 raft_consensus.cc:2258] T f8f620ccefff4fa580ad1a1d09c00b59 P 30acad1886794455a2b5617ba6da9aa1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:02.739395 11466 ts_tablet_manager.cc:1875] T f8f620ccefff4fa580ad1a1d09c00b59 P 30acad1886794455a2b5617ba6da9aa1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:56:02.798275 10209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:02.799551 9675 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:02.801223 9807 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:02.801263 9947 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:02.803625 10078 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:02.810782 8536 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:56:02.811007 8666 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:56:02.834800 11466 ts_tablet_manager.cc:1888] T f8f620ccefff4fa580ad1a1d09c00b59 P 30acad1886794455a2b5617ba6da9aa1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 13:56:02.834928 11466 log.cc:1196] T f8f620ccefff4fa580ad1a1d09c00b59 P 30acad1886794455a2b5617ba6da9aa1: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/wal/wals/f8f620ccefff4fa580ad1a1d09c00b59
I20240503 13:56:02.835610 8400 catalog_manager.cc:4830] TS 30acad1886794455a2b5617ba6da9aa1 (127.6.98.66:39533): tablet f8f620ccefff4fa580ad1a1d09c00b59 (table rebalance_test_table_2 [id=9453dfecf398486c84bd44eb1a914dc7]) successfully deleted
I20240503 13:56:02.871363 9967 consensus_queue.cc:237] T 8316d131c9ec41e39341f2c1a3c79722 P 6a0591a0f0b64c418af287c96b5b47b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6a0591a0f0b64c418af287c96b5b47b8" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 33575 } }
I20240503 13:56:02.872656 10874 raft_consensus.cc:2932] T 8316d131c9ec41e39341f2c1a3c79722 P 6a0591a0f0b64c418af287c96b5b47b8 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER a7346c212aa54138970650af7f721d0e (127.6.98.65) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6a0591a0f0b64c418af287c96b5b47b8" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 33575 } } }
I20240503 13:56:02.878856 8556 raft_consensus.cc:1261] T a9cea7e1421c43fb85282e212e39715d P a7346c212aa54138970650af7f721d0e [term 2 FOLLOWER]: Refusing update from remote peer 45bdf7ba415141cd85008dca119d1cf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 13:56:02.879343 11315 consensus_queue.cc:1035] T a9cea7e1421c43fb85282e212e39715d P 45bdf7ba415141cd85008dca119d1cf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:02.935797 8412 catalog_manager.cc:5484] T 8316d131c9ec41e39341f2c1a3c79722 P 6a0591a0f0b64c418af287c96b5b47b8 reported cstate change: config changed from index 4 to 6, VOTER a7346c212aa54138970650af7f721d0e (127.6.98.65) evicted. New cstate: current_term: 2 leader_uuid: "6a0591a0f0b64c418af287c96b5b47b8" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6a0591a0f0b64c418af287c96b5b47b8" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 33575 } } }
I20240503 13:56:02.939028 8536 tablet_service.cc:1514] Processing DeleteTablet for tablet 8316d131c9ec41e39341f2c1a3c79722 with delete_type TABLET_DATA_TOMBSTONED (TS a7346c212aa54138970650af7f721d0e not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:43290
I20240503 13:56:02.939535 11523 tablet_replica.cc:330] T 8316d131c9ec41e39341f2c1a3c79722 P a7346c212aa54138970650af7f721d0e: stopping tablet replica
I20240503 13:56:02.939818 11523 raft_consensus.cc:2229] T 8316d131c9ec41e39341f2c1a3c79722 P a7346c212aa54138970650af7f721d0e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:02.939977 11523 raft_consensus.cc:2258] T 8316d131c9ec41e39341f2c1a3c79722 P a7346c212aa54138970650af7f721d0e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:02.940946 11523 ts_tablet_manager.cc:1875] T 8316d131c9ec41e39341f2c1a3c79722 P a7346c212aa54138970650af7f721d0e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:56:03.005777 8666 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:56:03.006184 10078 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:03.007370 9807 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:03.007761 9947 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:03.009060 10209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:03.018748 9675 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:03.025158 8536 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:56:03.040408 11523 ts_tablet_manager.cc:1888] T 8316d131c9ec41e39341f2c1a3c79722 P a7346c212aa54138970650af7f721d0e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 13:56:03.040563 11523 log.cc:1196] T 8316d131c9ec41e39341f2c1a3c79722 P a7346c212aa54138970650af7f721d0e: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal/wals/8316d131c9ec41e39341f2c1a3c79722
I20240503 13:56:03.041249 8399 catalog_manager.cc:4830] TS a7346c212aa54138970650af7f721d0e (127.6.98.65:37097): tablet 8316d131c9ec41e39341f2c1a3c79722 (table rebalance_test_table_2 [id=9453dfecf398486c84bd44eb1a914dc7]) successfully deleted
I20240503 13:56:03.083869 10228 consensus_queue.cc:237] T b20e8dd34a25448a88f0658028fad27a P 44f093b99a6a403f8e092124ba9fea4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "44f093b99a6a403f8e092124ba9fea4c" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 39765 } }
I20240503 13:56:03.084944 11356 raft_consensus.cc:2932] T b20e8dd34a25448a88f0658028fad27a P 44f093b99a6a403f8e092124ba9fea4c [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER a7346c212aa54138970650af7f721d0e (127.6.98.65) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "44f093b99a6a403f8e092124ba9fea4c" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 39765 } } }
I20240503 13:56:03.134711 8406 catalog_manager.cc:5484] T b20e8dd34a25448a88f0658028fad27a P 44f093b99a6a403f8e092124ba9fea4c reported cstate change: config changed from index 4 to 6, VOTER a7346c212aa54138970650af7f721d0e (127.6.98.65) evicted. New cstate: current_term: 2 leader_uuid: "44f093b99a6a403f8e092124ba9fea4c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "44f093b99a6a403f8e092124ba9fea4c" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 39765 } } }
I20240503 13:56:03.138054 8536 tablet_service.cc:1514] Processing DeleteTablet for tablet b20e8dd34a25448a88f0658028fad27a with delete_type TABLET_DATA_TOMBSTONED (TS a7346c212aa54138970650af7f721d0e not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:43290
I20240503 13:56:03.138334 11523 tablet_replica.cc:330] T b20e8dd34a25448a88f0658028fad27a P a7346c212aa54138970650af7f721d0e: stopping tablet replica
I20240503 13:56:03.138480 11523 raft_consensus.cc:2229] T b20e8dd34a25448a88f0658028fad27a P a7346c212aa54138970650af7f721d0e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:03.138599 11523 raft_consensus.cc:2258] T b20e8dd34a25448a88f0658028fad27a P a7346c212aa54138970650af7f721d0e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:03.139247 11523 ts_tablet_manager.cc:1875] T b20e8dd34a25448a88f0658028fad27a P a7346c212aa54138970650af7f721d0e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:56:03.199077 10209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:03.199474 8666 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:56:03.200083 9947 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:03.200366 9675 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:03.200843 8536 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:56:03.206298 10078 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:03.226712 9807 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:03.237386 11523 ts_tablet_manager.cc:1888] T b20e8dd34a25448a88f0658028fad27a P a7346c212aa54138970650af7f721d0e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 13:56:03.237504 11523 log.cc:1196] T b20e8dd34a25448a88f0658028fad27a P a7346c212aa54138970650af7f721d0e: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal/wals/b20e8dd34a25448a88f0658028fad27a
I20240503 13:56:03.238123 8399 catalog_manager.cc:4830] TS a7346c212aa54138970650af7f721d0e (127.6.98.65:37097): tablet b20e8dd34a25448a88f0658028fad27a (table rebalance_test_table_4 [id=e6161850f776466d9d873af926bce637]) successfully deleted
I20240503 13:56:03.267006 9967 consensus_queue.cc:237] T f1d7cdc915b3484fbfbc8d4e5ae952b5 P 6a0591a0f0b64c418af287c96b5b47b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6a0591a0f0b64c418af287c96b5b47b8" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 33575 } }
I20240503 13:56:03.267818 11517 raft_consensus.cc:2932] T f1d7cdc915b3484fbfbc8d4e5ae952b5 P 6a0591a0f0b64c418af287c96b5b47b8 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 30acad1886794455a2b5617ba6da9aa1 (127.6.98.66) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6a0591a0f0b64c418af287c96b5b47b8" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 33575 } } }
I20240503 13:56:03.330103 8411 catalog_manager.cc:5484] T f1d7cdc915b3484fbfbc8d4e5ae952b5 P 6a0591a0f0b64c418af287c96b5b47b8 reported cstate change: config changed from index 4 to 6, VOTER 30acad1886794455a2b5617ba6da9aa1 (127.6.98.66) evicted. New cstate: current_term: 2 leader_uuid: "6a0591a0f0b64c418af287c96b5b47b8" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6a0591a0f0b64c418af287c96b5b47b8" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 33575 } } }
I20240503 13:56:03.336110 8666 tablet_service.cc:1514] Processing DeleteTablet for tablet f1d7cdc915b3484fbfbc8d4e5ae952b5 with delete_type TABLET_DATA_TOMBSTONED (TS 30acad1886794455a2b5617ba6da9aa1 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:53700
I20240503 13:56:03.339208 11632 tablet_replica.cc:330] T f1d7cdc915b3484fbfbc8d4e5ae952b5 P 30acad1886794455a2b5617ba6da9aa1: stopping tablet replica
I20240503 13:56:03.339380 11632 raft_consensus.cc:2229] T f1d7cdc915b3484fbfbc8d4e5ae952b5 P 30acad1886794455a2b5617ba6da9aa1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:03.339533 11632 raft_consensus.cc:2258] T f1d7cdc915b3484fbfbc8d4e5ae952b5 P 30acad1886794455a2b5617ba6da9aa1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:03.340152 11632 ts_tablet_manager.cc:1875] T f1d7cdc915b3484fbfbc8d4e5ae952b5 P 30acad1886794455a2b5617ba6da9aa1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:56:03.394328 10209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:03.395040 8666 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:56:03.398300 9807 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:03.410406 9947 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:03.410794 9675 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:03.411245 10078 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:03.428017 8536 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:56:03.437613 11632 ts_tablet_manager.cc:1888] T f1d7cdc915b3484fbfbc8d4e5ae952b5 P 30acad1886794455a2b5617ba6da9aa1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 13:56:03.437754 11632 log.cc:1196] T f1d7cdc915b3484fbfbc8d4e5ae952b5 P 30acad1886794455a2b5617ba6da9aa1: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/wal/wals/f1d7cdc915b3484fbfbc8d4e5ae952b5
I20240503 13:56:03.438482 8400 catalog_manager.cc:4830] TS 30acad1886794455a2b5617ba6da9aa1 (127.6.98.66:39533): tablet f1d7cdc915b3484fbfbc8d4e5ae952b5 (table rebalance_test_table_3 [id=3b610e4dbe9a4185a49772042aaa4f55]) successfully deleted
I20240503 13:56:03.469161 9695 consensus_queue.cc:237] T 922727f062fc4d6fbb7662a020bc3b65 P 45bdf7ba415141cd85008dca119d1cf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "45bdf7ba415141cd85008dca119d1cf9" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 40171 } }
I20240503 13:56:03.470082 11313 raft_consensus.cc:2932] T 922727f062fc4d6fbb7662a020bc3b65 P 45bdf7ba415141cd85008dca119d1cf9 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER a7346c212aa54138970650af7f721d0e (127.6.98.65) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "45bdf7ba415141cd85008dca119d1cf9" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 40171 } } }
I20240503 13:56:03.531952 8408 catalog_manager.cc:5484] T 922727f062fc4d6fbb7662a020bc3b65 P 45bdf7ba415141cd85008dca119d1cf9 reported cstate change: config changed from index 4 to 6, VOTER a7346c212aa54138970650af7f721d0e (127.6.98.65) evicted. New cstate: current_term: 2 leader_uuid: "45bdf7ba415141cd85008dca119d1cf9" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "45bdf7ba415141cd85008dca119d1cf9" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 40171 } } }
I20240503 13:56:03.534981 8536 tablet_service.cc:1514] Processing DeleteTablet for tablet 922727f062fc4d6fbb7662a020bc3b65 with delete_type TABLET_DATA_TOMBSTONED (TS a7346c212aa54138970650af7f721d0e not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:43290
I20240503 13:56:03.535248 11523 tablet_replica.cc:330] T 922727f062fc4d6fbb7662a020bc3b65 P a7346c212aa54138970650af7f721d0e: stopping tablet replica
I20240503 13:56:03.535423 11523 raft_consensus.cc:2229] T 922727f062fc4d6fbb7662a020bc3b65 P a7346c212aa54138970650af7f721d0e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:03.535548 11523 raft_consensus.cc:2258] T 922727f062fc4d6fbb7662a020bc3b65 P a7346c212aa54138970650af7f721d0e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:03.536072 11523 ts_tablet_manager.cc:1875] T 922727f062fc4d6fbb7662a020bc3b65 P a7346c212aa54138970650af7f721d0e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:56:03.599515 9947 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:03.601657 8666 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:56:03.602211 10209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:03.603076 9675 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:03.611361 9807 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:03.612027 8536 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:56:03.620936 10078 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:03.669768 9827 consensus_queue.cc:237] T 13c5e1f8f25345cab11a6944eae35f5a P 91d1a791a7504f71a4785f6aefe26f1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "91d1a791a7504f71a4785f6aefe26f1b" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 40653 } }
I20240503 13:56:03.670593 11626 raft_consensus.cc:2932] T 13c5e1f8f25345cab11a6944eae35f5a P 91d1a791a7504f71a4785f6aefe26f1b [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 30acad1886794455a2b5617ba6da9aa1 (127.6.98.66) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "91d1a791a7504f71a4785f6aefe26f1b" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 40653 } } }
I20240503 13:56:03.726537 11523 ts_tablet_manager.cc:1888] T 922727f062fc4d6fbb7662a020bc3b65 P a7346c212aa54138970650af7f721d0e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 13:56:03.726653 11523 log.cc:1196] T 922727f062fc4d6fbb7662a020bc3b65 P a7346c212aa54138970650af7f721d0e: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal/wals/922727f062fc4d6fbb7662a020bc3b65
I20240503 13:56:03.727248 8399 catalog_manager.cc:4830] TS a7346c212aa54138970650af7f721d0e (127.6.98.65:37097): tablet 922727f062fc4d6fbb7662a020bc3b65 (table rebalance_test_table_4 [id=e6161850f776466d9d873af926bce637]) successfully deleted
I20240503 13:56:03.728088 8408 catalog_manager.cc:5484] T 13c5e1f8f25345cab11a6944eae35f5a P 91d1a791a7504f71a4785f6aefe26f1b reported cstate change: config changed from index 4 to 6, VOTER 30acad1886794455a2b5617ba6da9aa1 (127.6.98.66) evicted. New cstate: current_term: 2 leader_uuid: "91d1a791a7504f71a4785f6aefe26f1b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "91d1a791a7504f71a4785f6aefe26f1b" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 40653 } } }
I20240503 13:56:03.731210 8666 tablet_service.cc:1514] Processing DeleteTablet for tablet 13c5e1f8f25345cab11a6944eae35f5a with delete_type TABLET_DATA_TOMBSTONED (TS 30acad1886794455a2b5617ba6da9aa1 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:53700
I20240503 13:56:03.731441 11632 tablet_replica.cc:330] T 13c5e1f8f25345cab11a6944eae35f5a P 30acad1886794455a2b5617ba6da9aa1: stopping tablet replica
I20240503 13:56:03.731571 11632 raft_consensus.cc:2229] T 13c5e1f8f25345cab11a6944eae35f5a P 30acad1886794455a2b5617ba6da9aa1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:03.731691 11632 raft_consensus.cc:2258] T 13c5e1f8f25345cab11a6944eae35f5a P 30acad1886794455a2b5617ba6da9aa1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:03.732183 11632 ts_tablet_manager.cc:1875] T 13c5e1f8f25345cab11a6944eae35f5a P 30acad1886794455a2b5617ba6da9aa1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:56:03.792239 8666 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:56:03.793234 10209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:03.793257 9675 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:03.795184 9947 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:03.803138 9807 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:03.804139 10078 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:03.824064 8536 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:56:03.836326 11632 ts_tablet_manager.cc:1888] T 13c5e1f8f25345cab11a6944eae35f5a P 30acad1886794455a2b5617ba6da9aa1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 13:56:03.836447 11632 log.cc:1196] T 13c5e1f8f25345cab11a6944eae35f5a P 30acad1886794455a2b5617ba6da9aa1: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/wal/wals/13c5e1f8f25345cab11a6944eae35f5a
I20240503 13:56:03.837055 8400 catalog_manager.cc:4830] TS 30acad1886794455a2b5617ba6da9aa1 (127.6.98.66:39533): tablet 13c5e1f8f25345cab11a6944eae35f5a (table rebalance_test_table_1 [id=3b9a448f00ad4aa5b759a21ca4c91628]) successfully deleted
I20240503 13:56:03.862339 10228 consensus_queue.cc:237] T 30ab7f35240e47ffb4c95fc56dc48fd3 P 44f093b99a6a403f8e092124ba9fea4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "44f093b99a6a403f8e092124ba9fea4c" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 39765 } }
I20240503 13:56:03.863063 11572 raft_consensus.cc:2932] T 30ab7f35240e47ffb4c95fc56dc48fd3 P 44f093b99a6a403f8e092124ba9fea4c [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER a7346c212aa54138970650af7f721d0e (127.6.98.65) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "44f093b99a6a403f8e092124ba9fea4c" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 39765 } } }
I20240503 13:56:03.865469 8412 catalog_manager.cc:5484] T 30ab7f35240e47ffb4c95fc56dc48fd3 P 44f093b99a6a403f8e092124ba9fea4c reported cstate change: config changed from index 4 to 6, VOTER a7346c212aa54138970650af7f721d0e (127.6.98.65) evicted. New cstate: current_term: 2 leader_uuid: "44f093b99a6a403f8e092124ba9fea4c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "44f093b99a6a403f8e092124ba9fea4c" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 39765 } } }
I20240503 13:56:03.868541 8536 tablet_service.cc:1514] Processing DeleteTablet for tablet 30ab7f35240e47ffb4c95fc56dc48fd3 with delete_type TABLET_DATA_TOMBSTONED (TS a7346c212aa54138970650af7f721d0e not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:43290
I20240503 13:56:03.868764 11523 tablet_replica.cc:330] T 30ab7f35240e47ffb4c95fc56dc48fd3 P a7346c212aa54138970650af7f721d0e: stopping tablet replica
I20240503 13:56:03.868887 11523 raft_consensus.cc:2229] T 30ab7f35240e47ffb4c95fc56dc48fd3 P a7346c212aa54138970650af7f721d0e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:03.868993 11523 raft_consensus.cc:2258] T 30ab7f35240e47ffb4c95fc56dc48fd3 P a7346c212aa54138970650af7f721d0e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:03.869498 11523 ts_tablet_manager.cc:1875] T 30ab7f35240e47ffb4c95fc56dc48fd3 P a7346c212aa54138970650af7f721d0e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:56:03.871582 11523 ts_tablet_manager.cc:1888] T 30ab7f35240e47ffb4c95fc56dc48fd3 P a7346c212aa54138970650af7f721d0e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 13:56:03.871644 11523 log.cc:1196] T 30ab7f35240e47ffb4c95fc56dc48fd3 P a7346c212aa54138970650af7f721d0e: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal/wals/30ab7f35240e47ffb4c95fc56dc48fd3
I20240503 13:56:03.872138 8399 catalog_manager.cc:4830] TS a7346c212aa54138970650af7f721d0e (127.6.98.65:37097): tablet 30ab7f35240e47ffb4c95fc56dc48fd3 (table rebalance_test_table_0 [id=477d6119baf54f00be410eea6e66b45d]) successfully deleted
I20240503 13:56:03.927357 9675 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:03.929668 8666 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:56:03.933192 9947 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:03.934088 9807 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:03.941738 10209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:03.944128 8536 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:56:03.945560 10078 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:03.990218 10097 consensus_queue.cc:237] T 9864208496e248fdaf0ae116f3dbae6f P a075f746a2e04544931d75c3dc463ef3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a075f746a2e04544931d75c3dc463ef3" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 33391 } }
I20240503 13:56:03.990939 11747 raft_consensus.cc:2932] T 9864208496e248fdaf0ae116f3dbae6f P a075f746a2e04544931d75c3dc463ef3 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 30acad1886794455a2b5617ba6da9aa1 (127.6.98.66) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a075f746a2e04544931d75c3dc463ef3" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 33391 } } }
I20240503 13:56:03.993135 8410 catalog_manager.cc:5484] T 9864208496e248fdaf0ae116f3dbae6f P a075f746a2e04544931d75c3dc463ef3 reported cstate change: config changed from index 4 to 6, VOTER 30acad1886794455a2b5617ba6da9aa1 (127.6.98.66) evicted. New cstate: current_term: 2 leader_uuid: "a075f746a2e04544931d75c3dc463ef3" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a075f746a2e04544931d75c3dc463ef3" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 33391 } } }
I20240503 13:56:03.995633 8666 tablet_service.cc:1514] Processing DeleteTablet for tablet 9864208496e248fdaf0ae116f3dbae6f with delete_type TABLET_DATA_TOMBSTONED (TS 30acad1886794455a2b5617ba6da9aa1 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:53700
I20240503 13:56:03.995853 11632 tablet_replica.cc:330] T 9864208496e248fdaf0ae116f3dbae6f P 30acad1886794455a2b5617ba6da9aa1: stopping tablet replica
I20240503 13:56:03.996012 11632 raft_consensus.cc:2229] T 9864208496e248fdaf0ae116f3dbae6f P 30acad1886794455a2b5617ba6da9aa1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:03.996119 11632 raft_consensus.cc:2258] T 9864208496e248fdaf0ae116f3dbae6f P 30acad1886794455a2b5617ba6da9aa1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:03.996600 11632 ts_tablet_manager.cc:1875] T 9864208496e248fdaf0ae116f3dbae6f P 30acad1886794455a2b5617ba6da9aa1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:56:03.998343 11632 ts_tablet_manager.cc:1888] T 9864208496e248fdaf0ae116f3dbae6f P 30acad1886794455a2b5617ba6da9aa1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 13:56:03.998406 11632 log.cc:1196] T 9864208496e248fdaf0ae116f3dbae6f P 30acad1886794455a2b5617ba6da9aa1: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/wal/wals/9864208496e248fdaf0ae116f3dbae6f
I20240503 13:56:03.998836 8400 catalog_manager.cc:4830] TS 30acad1886794455a2b5617ba6da9aa1 (127.6.98.66:39533): tablet 9864208496e248fdaf0ae116f3dbae6f (table rebalance_test_table_3 [id=3b610e4dbe9a4185a49772042aaa4f55]) successfully deleted
I20240503 13:56:04.055651 9675 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:04.056285 8666 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:56:04.057454 9947 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:04.068473 9807 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:04.068506 10209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:04.069877 8536 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:56:04.077634 10078 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:04.117944 9695 consensus_queue.cc:237] T a9cea7e1421c43fb85282e212e39715d P 45bdf7ba415141cd85008dca119d1cf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "45bdf7ba415141cd85008dca119d1cf9" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 40171 } }
I20240503 13:56:04.118754 11313 raft_consensus.cc:2932] T a9cea7e1421c43fb85282e212e39715d P 45bdf7ba415141cd85008dca119d1cf9 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER a7346c212aa54138970650af7f721d0e (127.6.98.65) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "45bdf7ba415141cd85008dca119d1cf9" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 40171 } } }
I20240503 13:56:04.120993 8410 catalog_manager.cc:5484] T a9cea7e1421c43fb85282e212e39715d P 45bdf7ba415141cd85008dca119d1cf9 reported cstate change: config changed from index 4 to 6, VOTER a7346c212aa54138970650af7f721d0e (127.6.98.65) evicted. New cstate: current_term: 2 leader_uuid: "45bdf7ba415141cd85008dca119d1cf9" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "45bdf7ba415141cd85008dca119d1cf9" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 40171 } } }
I20240503 13:56:04.123325 8536 tablet_service.cc:1514] Processing DeleteTablet for tablet a9cea7e1421c43fb85282e212e39715d with delete_type TABLET_DATA_TOMBSTONED (TS a7346c212aa54138970650af7f721d0e not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:43290
I20240503 13:56:04.123551 11523 tablet_replica.cc:330] T a9cea7e1421c43fb85282e212e39715d P a7346c212aa54138970650af7f721d0e: stopping tablet replica
I20240503 13:56:04.123658 11523 raft_consensus.cc:2229] T a9cea7e1421c43fb85282e212e39715d P a7346c212aa54138970650af7f721d0e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:04.123781 11523 raft_consensus.cc:2258] T a9cea7e1421c43fb85282e212e39715d P a7346c212aa54138970650af7f721d0e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:04.124253 11523 ts_tablet_manager.cc:1875] T a9cea7e1421c43fb85282e212e39715d P a7346c212aa54138970650af7f721d0e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:56:04.126159 11523 ts_tablet_manager.cc:1888] T a9cea7e1421c43fb85282e212e39715d P a7346c212aa54138970650af7f721d0e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 13:56:04.126247 11523 log.cc:1196] T a9cea7e1421c43fb85282e212e39715d P a7346c212aa54138970650af7f721d0e: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal/wals/a9cea7e1421c43fb85282e212e39715d
I20240503 13:56:04.126713 8399 catalog_manager.cc:4830] TS a7346c212aa54138970650af7f721d0e (127.6.98.65:37097): tablet a9cea7e1421c43fb85282e212e39715d (table rebalance_test_table_0 [id=477d6119baf54f00be410eea6e66b45d]) successfully deleted
I20240503 13:56:04.432242 10209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:04.433920 9675 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:04.435410 9947 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:04.444696 8666 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:56:04.447031 8536 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:56:04.447067 9807 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:04.447155 10078 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:04.540585 9947 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:04.542402 8666 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:56:04.543552 10209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:04.546698 8536 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:56:04.566056 9807 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:04.566185 9675 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:04.567440 10078 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:04.806488 8666 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:56:04.806481 9675 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:04.806686 9947 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:04.808247 8536 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:56:04.811830 10209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:04.817682 9807 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:04.830314 10078 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:04.938791 9947 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:04.941818 10209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:04.942056 8666 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:56:04.948621 9675 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:04.950484 10078 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:04.955752 9807 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:04.971844 8536 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:56:05.069444 9947 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:05.069769 9675 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:05.070106 10209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:05.072362 8666 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:56:05.079236 8536 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:56:05.086249 9807 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:05.090165 10078 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
9facb48813f1421ca415565225ef9d8b | 127.6.98.126:36629 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.6.98.84:33479 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+---------+----------+----------------+-----------------
30acad1886794455a2b5617ba6da9aa1 | 127.6.98.66:39533 | HEALTHY | <none> | 3 | 0
44f093b99a6a403f8e092124ba9fea4c | 127.6.98.71:39765 | HEALTHY | <none> | 2 | 0
45bdf7ba415141cd85008dca119d1cf9 | 127.6.98.67:40171 | HEALTHY | <none> | 2 | 0
6a0591a0f0b64c418af287c96b5b47b8 | 127.6.98.69:33575 | HEALTHY | <none> | 2 | 0
91d1a791a7504f71a4785f6aefe26f1b | 127.6.98.68:40653 | HEALTHY | <none> | 2 | 0
a075f746a2e04544931d75c3dc463ef3 | 127.6.98.70:33391 | HEALTHY | <none> | 2 | 0
a7346c212aa54138970650af7f721d0e | 127.6.98.65:37097 | HEALTHY | <none> | 3 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.6.98.65 | experimental | 127.6.98.65:37097
local_ip_for_outbound_sockets | 127.6.98.66 | experimental | 127.6.98.66:39533
local_ip_for_outbound_sockets | 127.6.98.67 | experimental | 127.6.98.67:40171
local_ip_for_outbound_sockets | 127.6.98.68 | experimental | 127.6.98.68:40653
local_ip_for_outbound_sockets | 127.6.98.69 | experimental | 127.6.98.69:33575
local_ip_for_outbound_sockets | 127.6.98.70 | experimental | 127.6.98.70:33391
local_ip_for_outbound_sockets | 127.6.98.71 | experimental | 127.6.98.71:39765
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.6.98.65:37097
server_dump_info_path | /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.6.98.66:39533
server_dump_info_path | /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.6.98.67:40171
server_dump_info_path | /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.6.98.68:40653
server_dump_info_path | /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.6.98.69:33575
server_dump_info_path | /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.6.98.70:33391
server_dump_info_path | /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.6.98.71:39765
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.6.98.84:33479 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 16
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 13:56:05.118273 6537 log_verifier.cc:126] Checking tablet 093b4bbf68574f39aa3128baa4f9fc9a
I20240503 13:56:05.129740 6537 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 093b4bbf68574f39aa3128baa4f9fc9a
I20240503 13:56:05.129830 6537 log_verifier.cc:126] Checking tablet 13c5e1f8f25345cab11a6944eae35f5a
I20240503 13:56:05.138397 6537 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 13c5e1f8f25345cab11a6944eae35f5a
I20240503 13:56:05.138470 6537 log_verifier.cc:126] Checking tablet 30ab7f35240e47ffb4c95fc56dc48fd3
I20240503 13:56:05.146685 6537 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 30ab7f35240e47ffb4c95fc56dc48fd3
I20240503 13:56:05.146762 6537 log_verifier.cc:126] Checking tablet 331586b138a94590b0a06e6fec84bf4a
I20240503 13:56:05.154376 6537 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 331586b138a94590b0a06e6fec84bf4a
I20240503 13:56:05.154441 6537 log_verifier.cc:126] Checking tablet 6ad697cafa9c42fcbcf0d7b22fbcb6b4
I20240503 13:56:05.163324 6537 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 6ad697cafa9c42fcbcf0d7b22fbcb6b4
I20240503 13:56:05.163512 6537 log_verifier.cc:126] Checking tablet 74d4351322b4423d81178462b8286342
I20240503 13:56:05.172444 6537 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 74d4351322b4423d81178462b8286342
I20240503 13:56:05.172540 6537 log_verifier.cc:126] Checking tablet 8316d131c9ec41e39341f2c1a3c79722
I20240503 13:56:05.180986 6537 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 8316d131c9ec41e39341f2c1a3c79722
I20240503 13:56:05.181054 6537 log_verifier.cc:126] Checking tablet 922727f062fc4d6fbb7662a020bc3b65
I20240503 13:56:05.190448 6537 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 922727f062fc4d6fbb7662a020bc3b65
I20240503 13:56:05.190519 6537 log_verifier.cc:126] Checking tablet 9864208496e248fdaf0ae116f3dbae6f
I20240503 13:56:05.198987 6537 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 9864208496e248fdaf0ae116f3dbae6f
I20240503 13:56:05.199060 6537 log_verifier.cc:126] Checking tablet a9cea7e1421c43fb85282e212e39715d
I20240503 13:56:05.207592 6537 log_verifier.cc:177] Verified matching terms for 6 ops in tablet a9cea7e1421c43fb85282e212e39715d
I20240503 13:56:05.207659 6537 log_verifier.cc:126] Checking tablet b20e8dd34a25448a88f0658028fad27a
I20240503 13:56:05.215981 6537 log_verifier.cc:177] Verified matching terms for 6 ops in tablet b20e8dd34a25448a88f0658028fad27a
I20240503 13:56:05.216056 6537 log_verifier.cc:126] Checking tablet b77257c31b6246b7a2ed06eb58a7736b
I20240503 13:56:05.224117 6537 log_verifier.cc:177] Verified matching terms for 3 ops in tablet b77257c31b6246b7a2ed06eb58a7736b
I20240503 13:56:05.224184 6537 log_verifier.cc:126] Checking tablet f1d7cdc915b3484fbfbc8d4e5ae952b5
I20240503 13:56:05.232466 6537 log_verifier.cc:177] Verified matching terms for 6 ops in tablet f1d7cdc915b3484fbfbc8d4e5ae952b5
I20240503 13:56:05.232556 6537 log_verifier.cc:126] Checking tablet f8f620ccefff4fa580ad1a1d09c00b59
I20240503 13:56:05.240891 6537 log_verifier.cc:177] Verified matching terms for 6 ops in tablet f8f620ccefff4fa580ad1a1d09c00b59
I20240503 13:56:05.240964 6537 log_verifier.cc:126] Checking tablet fbadee5b855b49dca94cc11e67bb1d21
I20240503 13:56:05.249070 6537 log_verifier.cc:177] Verified matching terms for 3 ops in tablet fbadee5b855b49dca94cc11e67bb1d21
I20240503 13:56:05.249137 6537 log_verifier.cc:126] Checking tablet ff18150f2a6e4a07b4ae653bf899a8d0
I20240503 13:56:05.257122 6537 log_verifier.cc:177] Verified matching terms for 3 ops in tablet ff18150f2a6e4a07b4ae653bf899a8d0
I20240503 13:56:05.259459 6537 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
/tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.98.72:0
--local_ip_for_outbound_sockets=127.6.98.72
--webserver_interface=127.6.98.72
--webserver_port=0
--tserver_master_addrs=127.6.98.126:36629
--builtin_ntp_servers=127.6.98.84:33479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 13:56:05.372218 12115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:56:05.372548 12115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:56:05.372617 12115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:56:05.375865 12115 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 13:56:05.375945 12115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:56:05.376016 12115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.98.72
I20240503 13:56:05.379734 12115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.98.84:33479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.98.72:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.6.98.72
--webserver_port=0
--tserver_master_addrs=127.6.98.126:36629
--never_fsync=true
--heap_profile_path=/tmp/kudu.12115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.98.72
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:56:05.380682 12115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:56:05.381825 12115 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:56:05.390134 12123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:56:05.390720 12115 server_base.cc:1034] running on GCE node
W20240503 13:56:05.390158 12121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:56:05.390300 12120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:56:05.393776 12115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:56:05.394515 12115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:56:05.395692 12115 hybrid_clock.cc:648] HybridClock initialized: now 1714744565395674 us; error 47 us; skew 500 ppm
I20240503 13:56:05.397853 12115 webserver.cc:460] Webserver started at http://127.6.98.72:37641/ using document root <none> and password file <none>
I20240503 13:56:05.398447 12115 fs_manager.cc:359] Metadata directory not provided
I20240503 13:56:05.398509 12115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:56:05.398689 12115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:56:05.400391 12115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "bf05b149fd7747b39d338fecfc0642ae"
format_stamp: "Formatted at 2024-05-03 13:56:05 on dist-test-slave-zg9z"
I20240503 13:56:05.400935 12115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "bf05b149fd7747b39d338fecfc0642ae"
format_stamp: "Formatted at 2024-05-03 13:56:05 on dist-test-slave-zg9z"
I20240503 13:56:05.404451 12115 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 13:56:05.406901 12129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:56:05.408068 12115 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:56:05.408180 12115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "bf05b149fd7747b39d338fecfc0642ae"
format_stamp: "Formatted at 2024-05-03 13:56:05 on dist-test-slave-zg9z"
I20240503 13:56:05.408272 12115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:56:05.428570 12115 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:56:05.429288 12115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:56:05.429458 12115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:56:05.430037 12115 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:56:05.430992 12115 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:56:05.431041 12115 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:56:05.431093 12115 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:56:05.431121 12115 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:56:05.439394 12115 rpc_server.cc:307] RPC server started. Bound to: 127.6.98.72:40277
I20240503 13:56:05.439440 12242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.98.72:40277 every 8 connection(s)
I20240503 13:56:05.440305 12115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 13:56:05.445715 6537 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu as pid 12115
I20240503 13:56:05.445827 6537 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 13:56:05.449584 12243 heartbeater.cc:347] Connected to a master server at 127.6.98.126:36629
I20240503 13:56:05.449821 12243 heartbeater.cc:464] Registering TS with master...
I20240503 13:56:05.450300 12243 heartbeater.cc:511] Master 127.6.98.126:36629 requested a full tablet report, sending...
I20240503 13:56:05.451297 8406 ts_manager.cc:194] Registered new tserver with Master: bf05b149fd7747b39d338fecfc0642ae (127.6.98.72:40277)
I20240503 13:56:05.451871 8406 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.98.72:60987
I20240503 13:56:05.643095 10209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:05.644840 8666 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:56:05.646085 9947 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:05.648730 8536 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:56:05.649118 9807 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:05.658170 9675 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:05.659031 10078 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:05.660404 12177 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:56:05.782548 10209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:05.783035 8666 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:56:05.784163 9675 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:05.795625 9807 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:05.800007 8536 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:56:05.801932 12177 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:56:05.810361 10078 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:05.816681 9947 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:05.858029 8687 consensus_queue.cc:237] T ff18150f2a6e4a07b4ae653bf899a8d0 P 30acad1886794455a2b5617ba6da9aa1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } } peers { permanent_uuid: "bf05b149fd7747b39d338fecfc0642ae" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 40277 } }
W20240503 13:56:05.862921 8623 consensus_peers.cc:485] T ff18150f2a6e4a07b4ae653bf899a8d0 P 30acad1886794455a2b5617ba6da9aa1 -> Peer bf05b149fd7747b39d338fecfc0642ae (127.6.98.72:40277): Couldn't send request to peer bf05b149fd7747b39d338fecfc0642ae. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ff18150f2a6e4a07b4ae653bf899a8d0. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:06.285977 12349 ts_tablet_manager.cc:902] T ff18150f2a6e4a07b4ae653bf899a8d0 P bf05b149fd7747b39d338fecfc0642ae: Initiating tablet copy from peer 30acad1886794455a2b5617ba6da9aa1 (127.6.98.66:39533)
I20240503 13:56:06.286711 12349 tablet_copy_client.cc:308] T ff18150f2a6e4a07b4ae653bf899a8d0 P bf05b149fd7747b39d338fecfc0642ae: tablet copy: Beginning tablet copy session from remote peer at address 127.6.98.66:39533
I20240503 13:56:06.290443 8707 tablet_copy_service.cc:140] P 30acad1886794455a2b5617ba6da9aa1: Received BeginTabletCopySession request for tablet ff18150f2a6e4a07b4ae653bf899a8d0 from peer bf05b149fd7747b39d338fecfc0642ae ({username='slave'} at 127.6.98.72:44991)
I20240503 13:56:06.290546 8707 tablet_copy_service.cc:161] P 30acad1886794455a2b5617ba6da9aa1: Beginning new tablet copy session on tablet ff18150f2a6e4a07b4ae653bf899a8d0 from peer bf05b149fd7747b39d338fecfc0642ae at {username='slave'} at 127.6.98.72:44991: session id = bf05b149fd7747b39d338fecfc0642ae-ff18150f2a6e4a07b4ae653bf899a8d0
I20240503 13:56:06.291277 8707 tablet_copy_source_session.cc:205] T ff18150f2a6e4a07b4ae653bf899a8d0 P 30acad1886794455a2b5617ba6da9aa1: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:56:06.294324 12349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff18150f2a6e4a07b4ae653bf899a8d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:06.299425 12349 tablet_copy_client.cc:783] T ff18150f2a6e4a07b4ae653bf899a8d0 P bf05b149fd7747b39d338fecfc0642ae: tablet copy: Starting download of 0 data blocks...
I20240503 13:56:06.299641 12349 tablet_copy_client.cc:647] T ff18150f2a6e4a07b4ae653bf899a8d0 P bf05b149fd7747b39d338fecfc0642ae: tablet copy: Starting download of 1 WAL segments...
I20240503 13:56:06.301152 12349 tablet_copy_client.cc:523] T ff18150f2a6e4a07b4ae653bf899a8d0 P bf05b149fd7747b39d338fecfc0642ae: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:56:06.302821 12349 tablet_bootstrap.cc:492] T ff18150f2a6e4a07b4ae653bf899a8d0 P bf05b149fd7747b39d338fecfc0642ae: Bootstrap starting.
I20240503 13:56:06.308442 12349 log.cc:824] T ff18150f2a6e4a07b4ae653bf899a8d0 P bf05b149fd7747b39d338fecfc0642ae: Log is configured to *not* fsync() on all Append() calls
I20240503 13:56:06.328195 12349 tablet_bootstrap.cc:492] T ff18150f2a6e4a07b4ae653bf899a8d0 P bf05b149fd7747b39d338fecfc0642ae: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 13:56:06.328676 12349 tablet_bootstrap.cc:492] T ff18150f2a6e4a07b4ae653bf899a8d0 P bf05b149fd7747b39d338fecfc0642ae: Bootstrap complete.
I20240503 13:56:06.328997 12349 ts_tablet_manager.cc:1367] T ff18150f2a6e4a07b4ae653bf899a8d0 P bf05b149fd7747b39d338fecfc0642ae: Time spent bootstrapping tablet: real 0.026s user 0.023s sys 0.004s
I20240503 13:56:06.331038 12349 raft_consensus.cc:348] T ff18150f2a6e4a07b4ae653bf899a8d0 P bf05b149fd7747b39d338fecfc0642ae [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } }
I20240503 13:56:06.331622 12349 raft_consensus.cc:729] T ff18150f2a6e4a07b4ae653bf899a8d0 P bf05b149fd7747b39d338fecfc0642ae [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf05b149fd7747b39d338fecfc0642ae, State: Initialized, Role: FOLLOWER
I20240503 13:56:06.331934 12349 consensus_queue.cc:260] T ff18150f2a6e4a07b4ae653bf899a8d0 P bf05b149fd7747b39d338fecfc0642ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } } peers { permanent_uuid: "bf05b149fd7747b39d338fecfc0642ae" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 40277 } }
I20240503 13:56:06.332715 12243 heartbeater.cc:503] Master 127.6.98.126:36629 was elected leader, sending a full tablet report...
I20240503 13:56:06.332787 12349 ts_tablet_manager.cc:1398] T ff18150f2a6e4a07b4ae653bf899a8d0 P bf05b149fd7747b39d338fecfc0642ae: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240503 13:56:06.333586 8707 tablet_copy_service.cc:342] P 30acad1886794455a2b5617ba6da9aa1: Request end of tablet copy session bf05b149fd7747b39d338fecfc0642ae-ff18150f2a6e4a07b4ae653bf899a8d0 received from {username='slave'} at 127.6.98.72:44991
I20240503 13:56:06.333693 8707 tablet_copy_service.cc:433] P 30acad1886794455a2b5617ba6da9aa1: ending tablet copy session bf05b149fd7747b39d338fecfc0642ae-ff18150f2a6e4a07b4ae653bf899a8d0 on tablet ff18150f2a6e4a07b4ae653bf899a8d0 with peer bf05b149fd7747b39d338fecfc0642ae
I20240503 13:56:06.868892 12197 raft_consensus.cc:1203] T ff18150f2a6e4a07b4ae653bf899a8d0 P bf05b149fd7747b39d338fecfc0642ae [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:56:06.870575 12359 raft_consensus.cc:2932] T ff18150f2a6e4a07b4ae653bf899a8d0 P 30acad1886794455a2b5617ba6da9aa1 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER bf05b149fd7747b39d338fecfc0642ae (127.6.98.72) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } } peers { permanent_uuid: "bf05b149fd7747b39d338fecfc0642ae" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 40277 } } }
I20240503 13:56:06.871275 12197 raft_consensus.cc:2932] T ff18150f2a6e4a07b4ae653bf899a8d0 P bf05b149fd7747b39d338fecfc0642ae [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER bf05b149fd7747b39d338fecfc0642ae (127.6.98.72) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } } peers { permanent_uuid: "bf05b149fd7747b39d338fecfc0642ae" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 40277 } } }
I20240503 13:56:06.872447 12359 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:43340 (request call id 0) took 1014 ms. Trace:
I20240503 13:56:06.872606 12359 rpcz_store.cc:276] 0503 13:56:05.857654 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 13:56:05.857697 (+ 43us) service_pool.cc:225] Handling call
0503 13:56:05.858494 (+ 797us) log.cc:842] Serialized 254 byte log entry
0503 13:56:06.872417 (+1013923us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":46,"raft.queue_time_us":103,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":147,"thread_start_us":109,"threads_started":2,"wal-append.queue_time_us":165,"wal-append.run_cpu_time_us":389,"wal-append.run_wall_time_us":1000463}
I20240503 13:56:06.874554 8406 catalog_manager.cc:5484] T ff18150f2a6e4a07b4ae653bf899a8d0 P bf05b149fd7747b39d338fecfc0642ae reported cstate change: config changed from index -1 to 4, VOTER bf05b149fd7747b39d338fecfc0642ae (127.6.98.72) added. New cstate: current_term: 1 leader_uuid: "30acad1886794455a2b5617ba6da9aa1" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } } peers { permanent_uuid: "bf05b149fd7747b39d338fecfc0642ae" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 40277 } } }
I20240503 13:56:06.895788 8556 consensus_queue.cc:237] T 093b4bbf68574f39aa3128baa4f9fc9a P a7346c212aa54138970650af7f721d0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } } peers { permanent_uuid: "bf05b149fd7747b39d338fecfc0642ae" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 40277 } }
W20240503 13:56:06.900413 8492 consensus_peers.cc:485] T 093b4bbf68574f39aa3128baa4f9fc9a P a7346c212aa54138970650af7f721d0e -> Peer bf05b149fd7747b39d338fecfc0642ae (127.6.98.72:40277): Couldn't send request to peer bf05b149fd7747b39d338fecfc0642ae. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 093b4bbf68574f39aa3128baa4f9fc9a. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:07.466550 12388 ts_tablet_manager.cc:902] T 093b4bbf68574f39aa3128baa4f9fc9a P bf05b149fd7747b39d338fecfc0642ae: Initiating tablet copy from peer a7346c212aa54138970650af7f721d0e (127.6.98.65:37097)
I20240503 13:56:07.467092 12388 tablet_copy_client.cc:308] T 093b4bbf68574f39aa3128baa4f9fc9a P bf05b149fd7747b39d338fecfc0642ae: tablet copy: Beginning tablet copy session from remote peer at address 127.6.98.65:37097
I20240503 13:56:07.470909 8576 tablet_copy_service.cc:140] P a7346c212aa54138970650af7f721d0e: Received BeginTabletCopySession request for tablet 093b4bbf68574f39aa3128baa4f9fc9a from peer bf05b149fd7747b39d338fecfc0642ae ({username='slave'} at 127.6.98.72:34587)
I20240503 13:56:07.471055 8576 tablet_copy_service.cc:161] P a7346c212aa54138970650af7f721d0e: Beginning new tablet copy session on tablet 093b4bbf68574f39aa3128baa4f9fc9a from peer bf05b149fd7747b39d338fecfc0642ae at {username='slave'} at 127.6.98.72:34587: session id = bf05b149fd7747b39d338fecfc0642ae-093b4bbf68574f39aa3128baa4f9fc9a
I20240503 13:56:07.471859 8576 tablet_copy_source_session.cc:205] T 093b4bbf68574f39aa3128baa4f9fc9a P a7346c212aa54138970650af7f721d0e: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:56:07.472785 12388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 093b4bbf68574f39aa3128baa4f9fc9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:07.474521 12388 tablet_copy_client.cc:783] T 093b4bbf68574f39aa3128baa4f9fc9a P bf05b149fd7747b39d338fecfc0642ae: tablet copy: Starting download of 0 data blocks...
I20240503 13:56:07.474671 12388 tablet_copy_client.cc:647] T 093b4bbf68574f39aa3128baa4f9fc9a P bf05b149fd7747b39d338fecfc0642ae: tablet copy: Starting download of 1 WAL segments...
I20240503 13:56:07.475893 12388 tablet_copy_client.cc:523] T 093b4bbf68574f39aa3128baa4f9fc9a P bf05b149fd7747b39d338fecfc0642ae: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:56:07.477005 12388 tablet_bootstrap.cc:492] T 093b4bbf68574f39aa3128baa4f9fc9a P bf05b149fd7747b39d338fecfc0642ae: Bootstrap starting.
I20240503 13:56:07.494524 12388 tablet_bootstrap.cc:492] T 093b4bbf68574f39aa3128baa4f9fc9a P bf05b149fd7747b39d338fecfc0642ae: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 13:56:07.494825 12388 tablet_bootstrap.cc:492] T 093b4bbf68574f39aa3128baa4f9fc9a P bf05b149fd7747b39d338fecfc0642ae: Bootstrap complete.
I20240503 13:56:07.495004 12388 ts_tablet_manager.cc:1367] T 093b4bbf68574f39aa3128baa4f9fc9a P bf05b149fd7747b39d338fecfc0642ae: Time spent bootstrapping tablet: real 0.018s user 0.013s sys 0.003s
I20240503 13:56:07.495550 12388 raft_consensus.cc:348] T 093b4bbf68574f39aa3128baa4f9fc9a P bf05b149fd7747b39d338fecfc0642ae [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } }
I20240503 13:56:07.495784 12388 raft_consensus.cc:729] T 093b4bbf68574f39aa3128baa4f9fc9a P bf05b149fd7747b39d338fecfc0642ae [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf05b149fd7747b39d338fecfc0642ae, State: Initialized, Role: FOLLOWER
I20240503 13:56:07.495898 12388 consensus_queue.cc:260] T 093b4bbf68574f39aa3128baa4f9fc9a P bf05b149fd7747b39d338fecfc0642ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } } peers { permanent_uuid: "bf05b149fd7747b39d338fecfc0642ae" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 40277 } }
I20240503 13:56:07.496258 12388 ts_tablet_manager.cc:1398] T 093b4bbf68574f39aa3128baa4f9fc9a P bf05b149fd7747b39d338fecfc0642ae: Time spent starting tablet: real 0.001s user 0.003s sys 0.001s
I20240503 13:56:07.496981 8576 tablet_copy_service.cc:342] P a7346c212aa54138970650af7f721d0e: Request end of tablet copy session bf05b149fd7747b39d338fecfc0642ae-093b4bbf68574f39aa3128baa4f9fc9a received from {username='slave'} at 127.6.98.72:34587
I20240503 13:56:07.497066 8576 tablet_copy_service.cc:433] P a7346c212aa54138970650af7f721d0e: ending tablet copy session bf05b149fd7747b39d338fecfc0642ae-093b4bbf68574f39aa3128baa4f9fc9a on tablet 093b4bbf68574f39aa3128baa4f9fc9a with peer bf05b149fd7747b39d338fecfc0642ae
I20240503 13:56:07.978253 12197 raft_consensus.cc:1203] T 093b4bbf68574f39aa3128baa4f9fc9a P bf05b149fd7747b39d338fecfc0642ae [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:56:07.979216 12398 raft_consensus.cc:2932] T 093b4bbf68574f39aa3128baa4f9fc9a P a7346c212aa54138970650af7f721d0e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER bf05b149fd7747b39d338fecfc0642ae (127.6.98.72) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } } peers { permanent_uuid: "bf05b149fd7747b39d338fecfc0642ae" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 40277 } } }
I20240503 13:56:07.979671 12197 raft_consensus.cc:2932] T 093b4bbf68574f39aa3128baa4f9fc9a P bf05b149fd7747b39d338fecfc0642ae [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER bf05b149fd7747b39d338fecfc0642ae (127.6.98.72) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } } peers { permanent_uuid: "bf05b149fd7747b39d338fecfc0642ae" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 40277 } } }
I20240503 13:56:07.980816 12398 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:39428 (request call id 0) took 1085 ms. Trace:
I20240503 13:56:07.980893 12398 rpcz_store.cc:276] 0503 13:56:06.895418 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 13:56:06.895489 (+ 71us) service_pool.cc:225] Handling call
0503 13:56:06.896227 (+ 738us) log.cc:842] Serialized 254 byte log entry
0503 13:56:07.980797 (+1084570us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":49,"raft.queue_time_us":209,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":123,"thread_start_us":230,"threads_started":2,"wal-append.queue_time_us":165,"wal-append.run_cpu_time_us":360,"wal-append.run_wall_time_us":1000429}
I20240503 13:56:07.982311 8411 catalog_manager.cc:5484] T 093b4bbf68574f39aa3128baa4f9fc9a P bf05b149fd7747b39d338fecfc0642ae reported cstate change: config changed from index -1 to 4, VOTER bf05b149fd7747b39d338fecfc0642ae (127.6.98.72) added. New cstate: current_term: 1 leader_uuid: "a7346c212aa54138970650af7f721d0e" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } } peers { permanent_uuid: "bf05b149fd7747b39d338fecfc0642ae" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 40277 } } }
I20240503 13:56:08.042984 9807 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:08.045439 8666 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:56:08.045859 9675 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:08.056545 10078 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:08.057178 9947 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:08.058311 10209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:08.069643 12177 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:56:08.069685 8536 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:56:08.102236 8556 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "093b4bbf68574f39aa3128baa4f9fc9a"
dest_uuid: "a7346c212aa54138970650af7f721d0e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:41822
I20240503 13:56:08.102392 8556 raft_consensus.cc:595] T 093b4bbf68574f39aa3128baa4f9fc9a P a7346c212aa54138970650af7f721d0e [term 1 LEADER]: Received request to transfer leadership
I20240503 13:56:08.153507 10209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:08.155532 8666 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:56:08.163779 12177 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:56:08.169200 9675 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:08.169193 9947 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:08.171667 10078 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:08.172150 9807 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:08.185943 8536 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:56:08.266763 8666 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:56:08.270684 9675 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:08.272050 9947 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:08.273111 9807 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:08.277624 10209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:08.294716 12177 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:56:08.295848 8536 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:56:08.296684 10078 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:08.329399 8687 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ff18150f2a6e4a07b4ae653bf899a8d0"
dest_uuid: "30acad1886794455a2b5617ba6da9aa1"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35804
I20240503 13:56:08.329556 8687 raft_consensus.cc:595] T ff18150f2a6e4a07b4ae653bf899a8d0 P 30acad1886794455a2b5617ba6da9aa1 [term 1 LEADER]: Received request to transfer leadership
I20240503 13:56:08.382639 12566 raft_consensus.cc:979] T ff18150f2a6e4a07b4ae653bf899a8d0 P 30acad1886794455a2b5617ba6da9aa1: : Instructing follower bf05b149fd7747b39d338fecfc0642ae to start an election
I20240503 13:56:08.382752 12566 raft_consensus.cc:1067] T ff18150f2a6e4a07b4ae653bf899a8d0 P 30acad1886794455a2b5617ba6da9aa1 [term 1 LEADER]: Signalling peer bf05b149fd7747b39d338fecfc0642ae to start an election
I20240503 13:56:08.383280 12197 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ff18150f2a6e4a07b4ae653bf899a8d0"
dest_uuid: "bf05b149fd7747b39d338fecfc0642ae"
from {username='slave'} at 127.6.98.66:54567
I20240503 13:56:08.383497 12197 raft_consensus.cc:482] T ff18150f2a6e4a07b4ae653bf899a8d0 P bf05b149fd7747b39d338fecfc0642ae [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 13:56:08.383613 12197 raft_consensus.cc:3037] T ff18150f2a6e4a07b4ae653bf899a8d0 P bf05b149fd7747b39d338fecfc0642ae [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:56:08.384775 12197 raft_consensus.cc:504] T ff18150f2a6e4a07b4ae653bf899a8d0 P bf05b149fd7747b39d338fecfc0642ae [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } } peers { permanent_uuid: "bf05b149fd7747b39d338fecfc0642ae" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 40277 } }
I20240503 13:56:08.385790 12197 leader_election.cc:288] T ff18150f2a6e4a07b4ae653bf899a8d0 P bf05b149fd7747b39d338fecfc0642ae [CANDIDATE]: Term 2 election: Requested vote from peers 30acad1886794455a2b5617ba6da9aa1 (127.6.98.66:39533)
I20240503 13:56:08.386428 8687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff18150f2a6e4a07b4ae653bf899a8d0" candidate_uuid: "bf05b149fd7747b39d338fecfc0642ae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "30acad1886794455a2b5617ba6da9aa1"
I20240503 13:56:08.387461 9947 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:08.387748 10209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:08.388469 8687 raft_consensus.cc:3032] T ff18150f2a6e4a07b4ae653bf899a8d0 P 30acad1886794455a2b5617ba6da9aa1 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 13:56:08.388568 8687 raft_consensus.cc:729] T ff18150f2a6e4a07b4ae653bf899a8d0 P 30acad1886794455a2b5617ba6da9aa1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 30acad1886794455a2b5617ba6da9aa1, State: Running, Role: LEADER
I20240503 13:56:08.388728 8687 consensus_queue.cc:260] T ff18150f2a6e4a07b4ae653bf899a8d0 P 30acad1886794455a2b5617ba6da9aa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } } peers { permanent_uuid: "bf05b149fd7747b39d338fecfc0642ae" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 40277 } }
I20240503 13:56:08.388892 8687 raft_consensus.cc:3037] T ff18150f2a6e4a07b4ae653bf899a8d0 P 30acad1886794455a2b5617ba6da9aa1 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:56:08.389968 8687 raft_consensus.cc:2454] T ff18150f2a6e4a07b4ae653bf899a8d0 P 30acad1886794455a2b5617ba6da9aa1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf05b149fd7747b39d338fecfc0642ae in term 2.
I20240503 13:56:08.390918 12132 leader_election.cc:302] T ff18150f2a6e4a07b4ae653bf899a8d0 P bf05b149fd7747b39d338fecfc0642ae [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 30acad1886794455a2b5617ba6da9aa1, bf05b149fd7747b39d338fecfc0642ae; no voters:
I20240503 13:56:08.391350 12396 raft_consensus.cc:2781] T ff18150f2a6e4a07b4ae653bf899a8d0 P bf05b149fd7747b39d338fecfc0642ae [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:56:08.391546 12396 raft_consensus.cc:686] T ff18150f2a6e4a07b4ae653bf899a8d0 P bf05b149fd7747b39d338fecfc0642ae [term 2 LEADER]: Becoming Leader. State: Replica: bf05b149fd7747b39d338fecfc0642ae, State: Running, Role: LEADER
I20240503 13:56:08.391870 12396 consensus_queue.cc:237] T ff18150f2a6e4a07b4ae653bf899a8d0 P bf05b149fd7747b39d338fecfc0642ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } } peers { permanent_uuid: "bf05b149fd7747b39d338fecfc0642ae" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 40277 } }
I20240503 13:56:08.395130 8406 catalog_manager.cc:5484] T ff18150f2a6e4a07b4ae653bf899a8d0 P bf05b149fd7747b39d338fecfc0642ae reported cstate change: term changed from 1 to 2, leader changed from 30acad1886794455a2b5617ba6da9aa1 (127.6.98.66) to bf05b149fd7747b39d338fecfc0642ae (127.6.98.72). New cstate: current_term: 2 leader_uuid: "bf05b149fd7747b39d338fecfc0642ae" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 } } peers { permanent_uuid: "bf05b149fd7747b39d338fecfc0642ae" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 40277 } } }
I20240503 13:56:08.397063 9675 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:08.401319 8666 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:08.403095 12177 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:56:08.409413 8536 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:56:08.422317 9807 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:08.435029 10078 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:08.525987 12570 raft_consensus.cc:979] T 093b4bbf68574f39aa3128baa4f9fc9a P a7346c212aa54138970650af7f721d0e: : Instructing follower bf05b149fd7747b39d338fecfc0642ae to start an election
I20240503 13:56:08.526111 12570 raft_consensus.cc:1067] T 093b4bbf68574f39aa3128baa4f9fc9a P a7346c212aa54138970650af7f721d0e [term 1 LEADER]: Signalling peer bf05b149fd7747b39d338fecfc0642ae to start an election
I20240503 13:56:08.526610 12197 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "093b4bbf68574f39aa3128baa4f9fc9a"
dest_uuid: "bf05b149fd7747b39d338fecfc0642ae"
from {username='slave'} at 127.6.98.65:39669
I20240503 13:56:08.526778 12197 raft_consensus.cc:482] T 093b4bbf68574f39aa3128baa4f9fc9a P bf05b149fd7747b39d338fecfc0642ae [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 13:56:08.526849 12197 raft_consensus.cc:3037] T 093b4bbf68574f39aa3128baa4f9fc9a P bf05b149fd7747b39d338fecfc0642ae [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:56:08.527788 12197 raft_consensus.cc:504] T 093b4bbf68574f39aa3128baa4f9fc9a P bf05b149fd7747b39d338fecfc0642ae [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } } peers { permanent_uuid: "bf05b149fd7747b39d338fecfc0642ae" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 40277 } }
I20240503 13:56:08.528179 12197 leader_election.cc:288] T 093b4bbf68574f39aa3128baa4f9fc9a P bf05b149fd7747b39d338fecfc0642ae [CANDIDATE]: Term 2 election: Requested vote from peers a7346c212aa54138970650af7f721d0e (127.6.98.65:37097)
I20240503 13:56:08.528661 8556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "093b4bbf68574f39aa3128baa4f9fc9a" candidate_uuid: "bf05b149fd7747b39d338fecfc0642ae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "a7346c212aa54138970650af7f721d0e"
I20240503 13:56:08.528787 8556 raft_consensus.cc:3032] T 093b4bbf68574f39aa3128baa4f9fc9a P a7346c212aa54138970650af7f721d0e [term 1 LEADER]: Stepping down as leader of term 1
I20240503 13:56:08.528817 8556 raft_consensus.cc:729] T 093b4bbf68574f39aa3128baa4f9fc9a P a7346c212aa54138970650af7f721d0e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a7346c212aa54138970650af7f721d0e, State: Running, Role: LEADER
I20240503 13:56:08.528980 8556 consensus_queue.cc:260] T 093b4bbf68574f39aa3128baa4f9fc9a P a7346c212aa54138970650af7f721d0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } } peers { permanent_uuid: "bf05b149fd7747b39d338fecfc0642ae" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 40277 } }
I20240503 13:56:08.529098 8556 raft_consensus.cc:3037] T 093b4bbf68574f39aa3128baa4f9fc9a P a7346c212aa54138970650af7f721d0e [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:56:08.529821 8556 raft_consensus.cc:2454] T 093b4bbf68574f39aa3128baa4f9fc9a P a7346c212aa54138970650af7f721d0e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf05b149fd7747b39d338fecfc0642ae in term 2.
I20240503 13:56:08.530176 12131 leader_election.cc:302] T 093b4bbf68574f39aa3128baa4f9fc9a P bf05b149fd7747b39d338fecfc0642ae [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: a7346c212aa54138970650af7f721d0e, bf05b149fd7747b39d338fecfc0642ae; no voters:
I20240503 13:56:08.530377 12568 raft_consensus.cc:2781] T 093b4bbf68574f39aa3128baa4f9fc9a P bf05b149fd7747b39d338fecfc0642ae [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:56:08.530458 12568 raft_consensus.cc:686] T 093b4bbf68574f39aa3128baa4f9fc9a P bf05b149fd7747b39d338fecfc0642ae [term 2 LEADER]: Becoming Leader. State: Replica: bf05b149fd7747b39d338fecfc0642ae, State: Running, Role: LEADER
I20240503 13:56:08.530596 12568 consensus_queue.cc:237] T 093b4bbf68574f39aa3128baa4f9fc9a P bf05b149fd7747b39d338fecfc0642ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } } peers { permanent_uuid: "bf05b149fd7747b39d338fecfc0642ae" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 40277 } }
I20240503 13:56:08.532125 8406 catalog_manager.cc:5484] T 093b4bbf68574f39aa3128baa4f9fc9a P bf05b149fd7747b39d338fecfc0642ae reported cstate change: term changed from 1 to 2, leader changed from a7346c212aa54138970650af7f721d0e (127.6.98.65) to bf05b149fd7747b39d338fecfc0642ae (127.6.98.72). New cstate: current_term: 2 leader_uuid: "bf05b149fd7747b39d338fecfc0642ae" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 } } peers { permanent_uuid: "bf05b149fd7747b39d338fecfc0642ae" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 40277 } } }
I20240503 13:56:08.714144 9947 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:08.714689 8666 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:08.715773 9675 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:08.720916 10209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:08.723479 8536 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:08.730417 10078 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:08.740653 9807 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:08.755438 12177 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:08.833982 9675 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:08.834086 10209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:08.838383 12177 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:08.842468 10078 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:08.842861 9947 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:08.844290 8536 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:08.845850 9807 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:08.857082 8666 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:08.872315 8687 raft_consensus.cc:1261] T ff18150f2a6e4a07b4ae653bf899a8d0 P 30acad1886794455a2b5617ba6da9aa1 [term 2 FOLLOWER]: Refusing update from remote peer bf05b149fd7747b39d338fecfc0642ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 13:56:08.873001 12568 consensus_queue.cc:1035] T ff18150f2a6e4a07b4ae653bf899a8d0 P bf05b149fd7747b39d338fecfc0642ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "30acad1886794455a2b5617ba6da9aa1" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 39533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:09.055524 8556 raft_consensus.cc:1261] T 093b4bbf68574f39aa3128baa4f9fc9a P a7346c212aa54138970650af7f721d0e [term 2 FOLLOWER]: Refusing update from remote peer bf05b149fd7747b39d338fecfc0642ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 13:56:09.056044 12396 consensus_queue.cc:1035] T 093b4bbf68574f39aa3128baa4f9fc9a P bf05b149fd7747b39d338fecfc0642ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7346c212aa54138970650af7f721d0e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:09.146340 10209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:09.148176 9675 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:09.153532 9807 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:09.154433 12177 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:09.156543 8536 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:09.167657 10078 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:09.168728 9947 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:09.191031 8666 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:09.227520 12197 consensus_queue.cc:237] T 093b4bbf68574f39aa3128baa4f9fc9a P bf05b149fd7747b39d338fecfc0642ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bf05b149fd7747b39d338fecfc0642ae" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 40277 } }
I20240503 13:56:09.228204 12568 raft_consensus.cc:2932] T 093b4bbf68574f39aa3128baa4f9fc9a P bf05b149fd7747b39d338fecfc0642ae [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER a7346c212aa54138970650af7f721d0e (127.6.98.65) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bf05b149fd7747b39d338fecfc0642ae" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 40277 } } }
I20240503 13:56:09.230593 8406 catalog_manager.cc:5484] T 093b4bbf68574f39aa3128baa4f9fc9a P bf05b149fd7747b39d338fecfc0642ae reported cstate change: config changed from index 4 to 6, VOTER a7346c212aa54138970650af7f721d0e (127.6.98.65) evicted. New cstate: current_term: 2 leader_uuid: "bf05b149fd7747b39d338fecfc0642ae" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bf05b149fd7747b39d338fecfc0642ae" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 40277 } } }
I20240503 13:56:09.233655 8536 tablet_service.cc:1514] Processing DeleteTablet for tablet 093b4bbf68574f39aa3128baa4f9fc9a with delete_type TABLET_DATA_TOMBSTONED (TS a7346c212aa54138970650af7f721d0e not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:43290
I20240503 13:56:09.234035 12715 tablet_replica.cc:330] T 093b4bbf68574f39aa3128baa4f9fc9a P a7346c212aa54138970650af7f721d0e: stopping tablet replica
I20240503 13:56:09.234170 12715 raft_consensus.cc:2229] T 093b4bbf68574f39aa3128baa4f9fc9a P a7346c212aa54138970650af7f721d0e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:09.234285 12715 raft_consensus.cc:2258] T 093b4bbf68574f39aa3128baa4f9fc9a P a7346c212aa54138970650af7f721d0e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:09.235792 12715 ts_tablet_manager.cc:1875] T 093b4bbf68574f39aa3128baa4f9fc9a P a7346c212aa54138970650af7f721d0e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:56:09.238346 12715 ts_tablet_manager.cc:1888] T 093b4bbf68574f39aa3128baa4f9fc9a P a7346c212aa54138970650af7f721d0e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 13:56:09.238418 12715 log.cc:1196] T 093b4bbf68574f39aa3128baa4f9fc9a P a7346c212aa54138970650af7f721d0e: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal/wals/093b4bbf68574f39aa3128baa4f9fc9a
I20240503 13:56:09.238951 8399 catalog_manager.cc:4830] TS a7346c212aa54138970650af7f721d0e (127.6.98.65:37097): tablet 093b4bbf68574f39aa3128baa4f9fc9a (table rebalance_test_table_2 [id=9453dfecf398486c84bd44eb1a914dc7]) successfully deleted
I20240503 13:56:09.288991 10209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:09.291707 8666 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:09.292856 9947 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:09.297698 9807 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:09.307467 12177 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:09.309396 10078 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:09.313077 9675 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:09.321005 8536 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:09.363270 12197 consensus_queue.cc:237] T ff18150f2a6e4a07b4ae653bf899a8d0 P bf05b149fd7747b39d338fecfc0642ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bf05b149fd7747b39d338fecfc0642ae" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 40277 } }
I20240503 13:56:09.363911 12568 raft_consensus.cc:2932] T ff18150f2a6e4a07b4ae653bf899a8d0 P bf05b149fd7747b39d338fecfc0642ae [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 30acad1886794455a2b5617ba6da9aa1 (127.6.98.66) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bf05b149fd7747b39d338fecfc0642ae" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 40277 } } }
I20240503 13:56:09.366094 8410 catalog_manager.cc:5484] T ff18150f2a6e4a07b4ae653bf899a8d0 P bf05b149fd7747b39d338fecfc0642ae reported cstate change: config changed from index 4 to 6, VOTER 30acad1886794455a2b5617ba6da9aa1 (127.6.98.66) evicted. New cstate: current_term: 2 leader_uuid: "bf05b149fd7747b39d338fecfc0642ae" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bf05b149fd7747b39d338fecfc0642ae" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 40277 } } }
I20240503 13:56:09.368875 8666 tablet_service.cc:1514] Processing DeleteTablet for tablet ff18150f2a6e4a07b4ae653bf899a8d0 with delete_type TABLET_DATA_TOMBSTONED (TS 30acad1886794455a2b5617ba6da9aa1 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:53700
I20240503 13:56:09.369233 12764 tablet_replica.cc:330] T ff18150f2a6e4a07b4ae653bf899a8d0 P 30acad1886794455a2b5617ba6da9aa1: stopping tablet replica
I20240503 13:56:09.369359 12764 raft_consensus.cc:2229] T ff18150f2a6e4a07b4ae653bf899a8d0 P 30acad1886794455a2b5617ba6da9aa1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:09.369469 12764 raft_consensus.cc:2258] T ff18150f2a6e4a07b4ae653bf899a8d0 P 30acad1886794455a2b5617ba6da9aa1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:09.370651 12764 ts_tablet_manager.cc:1875] T ff18150f2a6e4a07b4ae653bf899a8d0 P 30acad1886794455a2b5617ba6da9aa1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:56:09.372570 12764 ts_tablet_manager.cc:1888] T ff18150f2a6e4a07b4ae653bf899a8d0 P 30acad1886794455a2b5617ba6da9aa1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 13:56:09.372637 12764 log.cc:1196] T ff18150f2a6e4a07b4ae653bf899a8d0 P 30acad1886794455a2b5617ba6da9aa1: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/wal/wals/ff18150f2a6e4a07b4ae653bf899a8d0
I20240503 13:56:09.373137 8400 catalog_manager.cc:4830] TS 30acad1886794455a2b5617ba6da9aa1 (127.6.98.66:39533): tablet ff18150f2a6e4a07b4ae653bf899a8d0 (table rebalance_test_table_4 [id=e6161850f776466d9d873af926bce637]) successfully deleted
I20240503 13:56:09.652459 10209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:09.652583 8666 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:09.655272 9947 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:09.665171 8536 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:09.665395 9675 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:09.665647 9807 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:09.666862 10078 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:09.683396 12177 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:09.772280 9947 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:09.774163 10209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:09.775197 8666 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:09.779970 8536 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:09.785259 9675 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:09.793313 9807 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:09.794858 10078 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:09.807559 12177 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:09.893474 10209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:09.897351 8666 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:09.901994 8536 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:09.903852 9675 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:09.910079 9947 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:09.912061 10078 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:09.916167 9807 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:09.928298 12177 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
9facb48813f1421ca415565225ef9d8b | 127.6.98.126:36629 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.6.98.84:33479 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+---------+----------+----------------+-----------------
30acad1886794455a2b5617ba6da9aa1 | 127.6.98.66:39533 | HEALTHY | <none> | 2 | 0
44f093b99a6a403f8e092124ba9fea4c | 127.6.98.71:39765 | HEALTHY | <none> | 2 | 0
45bdf7ba415141cd85008dca119d1cf9 | 127.6.98.67:40171 | HEALTHY | <none> | 2 | 0
6a0591a0f0b64c418af287c96b5b47b8 | 127.6.98.69:33575 | HEALTHY | <none> | 2 | 0
91d1a791a7504f71a4785f6aefe26f1b | 127.6.98.68:40653 | HEALTHY | <none> | 2 | 0
a075f746a2e04544931d75c3dc463ef3 | 127.6.98.70:33391 | HEALTHY | <none> | 2 | 0
a7346c212aa54138970650af7f721d0e | 127.6.98.65:37097 | HEALTHY | <none> | 2 | 0
bf05b149fd7747b39d338fecfc0642ae | 127.6.98.72:40277 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.6.98.65 | experimental | 127.6.98.65:37097
local_ip_for_outbound_sockets | 127.6.98.66 | experimental | 127.6.98.66:39533
local_ip_for_outbound_sockets | 127.6.98.67 | experimental | 127.6.98.67:40171
local_ip_for_outbound_sockets | 127.6.98.68 | experimental | 127.6.98.68:40653
local_ip_for_outbound_sockets | 127.6.98.69 | experimental | 127.6.98.69:33575
local_ip_for_outbound_sockets | 127.6.98.70 | experimental | 127.6.98.70:33391
local_ip_for_outbound_sockets | 127.6.98.71 | experimental | 127.6.98.71:39765
local_ip_for_outbound_sockets | 127.6.98.72 | experimental | 127.6.98.72:40277
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.6.98.65:37097
server_dump_info_path | /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.6.98.66:39533
server_dump_info_path | /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.6.98.67:40171
server_dump_info_path | /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.6.98.68:40653
server_dump_info_path | /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.6.98.69:33575
server_dump_info_path | /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.6.98.70:33391
server_dump_info_path | /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.6.98.71:39765
server_dump_info_path | /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.6.98.72:40277
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.6.98.84:33479 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 2
Maximum | 2
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 16
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 13:56:09.952929 6537 log_verifier.cc:126] Checking tablet 093b4bbf68574f39aa3128baa4f9fc9a
I20240503 13:56:09.961455 6537 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 093b4bbf68574f39aa3128baa4f9fc9a
I20240503 13:56:09.961531 6537 log_verifier.cc:126] Checking tablet 13c5e1f8f25345cab11a6944eae35f5a
I20240503 13:56:09.967525 6537 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 13c5e1f8f25345cab11a6944eae35f5a
I20240503 13:56:09.967592 6537 log_verifier.cc:126] Checking tablet 30ab7f35240e47ffb4c95fc56dc48fd3
I20240503 13:56:09.973189 6537 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 30ab7f35240e47ffb4c95fc56dc48fd3
I20240503 13:56:09.973246 6537 log_verifier.cc:126] Checking tablet 331586b138a94590b0a06e6fec84bf4a
I20240503 13:56:09.977669 6537 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 331586b138a94590b0a06e6fec84bf4a
I20240503 13:56:09.977726 6537 log_verifier.cc:126] Checking tablet 6ad697cafa9c42fcbcf0d7b22fbcb6b4
I20240503 13:56:09.983093 6537 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 6ad697cafa9c42fcbcf0d7b22fbcb6b4
I20240503 13:56:09.983150 6537 log_verifier.cc:126] Checking tablet 74d4351322b4423d81178462b8286342
I20240503 13:56:09.988605 6537 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 74d4351322b4423d81178462b8286342
I20240503 13:56:09.988662 6537 log_verifier.cc:126] Checking tablet 8316d131c9ec41e39341f2c1a3c79722
I20240503 13:56:09.994339 6537 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 8316d131c9ec41e39341f2c1a3c79722
I20240503 13:56:09.994397 6537 log_verifier.cc:126] Checking tablet 922727f062fc4d6fbb7662a020bc3b65
I20240503 13:56:09.999940 6537 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 922727f062fc4d6fbb7662a020bc3b65
I20240503 13:56:10.000005 6537 log_verifier.cc:126] Checking tablet 9864208496e248fdaf0ae116f3dbae6f
I20240503 13:56:10.005796 6537 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 9864208496e248fdaf0ae116f3dbae6f
I20240503 13:56:10.005858 6537 log_verifier.cc:126] Checking tablet a9cea7e1421c43fb85282e212e39715d
I20240503 13:56:10.011672 6537 log_verifier.cc:177] Verified matching terms for 6 ops in tablet a9cea7e1421c43fb85282e212e39715d
I20240503 13:56:10.011737 6537 log_verifier.cc:126] Checking tablet b20e8dd34a25448a88f0658028fad27a
I20240503 13:56:10.017473 6537 log_verifier.cc:177] Verified matching terms for 6 ops in tablet b20e8dd34a25448a88f0658028fad27a
I20240503 13:56:10.017534 6537 log_verifier.cc:126] Checking tablet b77257c31b6246b7a2ed06eb58a7736b
I20240503 13:56:10.022864 6537 log_verifier.cc:177] Verified matching terms for 3 ops in tablet b77257c31b6246b7a2ed06eb58a7736b
I20240503 13:56:10.022925 6537 log_verifier.cc:126] Checking tablet f1d7cdc915b3484fbfbc8d4e5ae952b5
I20240503 13:56:10.028656 6537 log_verifier.cc:177] Verified matching terms for 6 ops in tablet f1d7cdc915b3484fbfbc8d4e5ae952b5
I20240503 13:56:10.028712 6537 log_verifier.cc:126] Checking tablet f8f620ccefff4fa580ad1a1d09c00b59
I20240503 13:56:10.035735 6537 log_verifier.cc:177] Verified matching terms for 6 ops in tablet f8f620ccefff4fa580ad1a1d09c00b59
I20240503 13:56:10.035830 6537 log_verifier.cc:126] Checking tablet fbadee5b855b49dca94cc11e67bb1d21
I20240503 13:56:10.041530 6537 log_verifier.cc:177] Verified matching terms for 3 ops in tablet fbadee5b855b49dca94cc11e67bb1d21
I20240503 13:56:10.041608 6537 log_verifier.cc:126] Checking tablet ff18150f2a6e4a07b4ae653bf899a8d0
I20240503 13:56:10.050315 6537 log_verifier.cc:177] Verified matching terms for 6 ops in tablet ff18150f2a6e4a07b4ae653bf899a8d0
I20240503 13:56:10.051096 6537 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu with pid 8474
I20240503 13:56:10.057577 6537 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu with pid 8605
I20240503 13:56:10.063511 6537 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu with pid 9613
I20240503 13:56:10.068904 6537 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu with pid 9744
I20240503 13:56:10.073917 6537 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu with pid 9877
I20240503 13:56:10.079056 6537 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu with pid 10016
I20240503 13:56:10.083946 6537 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu with pid 10147
I20240503 13:56:10.088842 6537 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu with pid 12115
I20240503 13:56:10.093582 6537 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu with pid 8382
2024-05-03T13:56:10Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/0 (27964 ms)
[----------] 1 test from RebalanceParamTest (27964 ms total)
[----------] 1 test from DDLDuringRebalancingTest
[ RUN ] DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing/0
I20240503 13:56:10.128621 6537 test_util.cc:268] Using random seed: 352037096
I20240503 13:56:10.129388 6537 ts_itest-base.cc:112] Starting cluster with:
I20240503 13:56:10.129436 6537 ts_itest-base.cc:113] --------------
I20240503 13:56:10.129446 6537 ts_itest-base.cc:114] 8 tablet servers
I20240503 13:56:10.129454 6537 ts_itest-base.cc:115] 3 replicas per TS
I20240503 13:56:10.129463 6537 ts_itest-base.cc:116] --------------
2024-05-03T13:56:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T13:56:10Z Disabled control of system clock
I20240503 13:56:10.137439 6537 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
/tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.98.126:44613
--webserver_interface=127.6.98.126
--webserver_port=0
--builtin_ntp_servers=127.6.98.84:39449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.98.126:44613
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240503 13:56:10.246897 12881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:56:10.247133 12881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:56:10.247174 12881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:56:10.250314 12881 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 13:56:10.250384 12881 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 13:56:10.250407 12881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:56:10.250423 12881 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 13:56:10.250440 12881 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 13:56:10.254210 12881 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.98.84:39449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.6.98.126:44613
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.98.126:44613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.6.98.126
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.12881
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:56:10.255131 12881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:56:10.256286 12881 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:56:10.263221 12889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:56:10.263453 12886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:56:10.263590 12887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:56:10.263913 12881 server_base.cc:1034] running on GCE node
I20240503 13:56:10.264393 12881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:56:10.265327 12881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:56:10.266491 12881 hybrid_clock.cc:648] HybridClock initialized: now 1714744570266483 us; error 31 us; skew 500 ppm
I20240503 13:56:10.273324 12881 webserver.cc:460] Webserver started at http://127.6.98.126:37945/ using document root <none> and password file <none>
I20240503 13:56:10.273890 12881 fs_manager.cc:359] Metadata directory not provided
I20240503 13:56:10.273950 12881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:56:10.274138 12881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:56:10.275797 12881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "f21c260fa03f4452870a21b3091e1130"
format_stamp: "Formatted at 2024-05-03 13:56:10 on dist-test-slave-zg9z"
I20240503 13:56:10.276273 12881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "f21c260fa03f4452870a21b3091e1130"
format_stamp: "Formatted at 2024-05-03 13:56:10 on dist-test-slave-zg9z"
I20240503 13:56:10.279600 12881 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240503 13:56:10.282037 12895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:56:10.283032 12881 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20240503 13:56:10.283161 12881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/master-0/wal
uuid: "f21c260fa03f4452870a21b3091e1130"
format_stamp: "Formatted at 2024-05-03 13:56:10 on dist-test-slave-zg9z"
I20240503 13:56:10.283248 12881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:56:10.294178 12881 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:56:10.294811 12881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:56:10.294974 12881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:56:10.301337 12881 rpc_server.cc:307] RPC server started. Bound to: 127.6.98.126:44613
I20240503 13:56:10.301347 12947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.98.126:44613 every 8 connection(s)
I20240503 13:56:10.302315 12881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 13:56:10.303054 6537 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu as pid 12881
I20240503 13:56:10.303179 6537 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 13:56:10.305393 12948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:10.312557 12948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Bootstrap starting.
I20240503 13:56:10.314921 12948 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:10.315618 12948 log.cc:824] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Log is configured to *not* fsync() on all Append() calls
I20240503 13:56:10.317271 12948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: No bootstrap required, opened a new log
I20240503 13:56:10.319764 12948 raft_consensus.cc:348] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f21c260fa03f4452870a21b3091e1130" member_type: VOTER last_known_addr { host: "127.6.98.126" port: 44613 } }
I20240503 13:56:10.319958 12948 raft_consensus.cc:374] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:10.319993 12948 raft_consensus.cc:729] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c260fa03f4452870a21b3091e1130, State: Initialized, Role: FOLLOWER
I20240503 13:56:10.320421 12948 consensus_queue.cc:260] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f21c260fa03f4452870a21b3091e1130" member_type: VOTER last_known_addr { host: "127.6.98.126" port: 44613 } }
I20240503 13:56:10.320576 12948 raft_consensus.cc:388] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:56:10.320639 12948 raft_consensus.cc:482] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:56:10.320735 12948 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:10.321494 12948 raft_consensus.cc:504] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f21c260fa03f4452870a21b3091e1130" member_type: VOTER last_known_addr { host: "127.6.98.126" port: 44613 } }
I20240503 13:56:10.321846 12948 leader_election.cc:302] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f21c260fa03f4452870a21b3091e1130; no voters:
I20240503 13:56:10.322167 12948 leader_election.cc:288] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 13:56:10.322479 12953 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:10.322759 12953 raft_consensus.cc:686] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130 [term 1 LEADER]: Becoming Leader. State: Replica: f21c260fa03f4452870a21b3091e1130, State: Running, Role: LEADER
I20240503 13:56:10.323064 12953 consensus_queue.cc:237] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f21c260fa03f4452870a21b3091e1130" member_type: VOTER last_known_addr { host: "127.6.98.126" port: 44613 } }
I20240503 13:56:10.323297 12948 sys_catalog.cc:564] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 13:56:10.324551 12954 sys_catalog.cc:455] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f21c260fa03f4452870a21b3091e1130" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f21c260fa03f4452870a21b3091e1130" member_type: VOTER last_known_addr { host: "127.6.98.126" port: 44613 } } }
I20240503 13:56:10.324558 12955 sys_catalog.cc:455] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f21c260fa03f4452870a21b3091e1130. Latest consensus state: current_term: 1 leader_uuid: "f21c260fa03f4452870a21b3091e1130" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f21c260fa03f4452870a21b3091e1130" member_type: VOTER last_known_addr { host: "127.6.98.126" port: 44613 } } }
I20240503 13:56:10.324726 12954 sys_catalog.cc:458] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130 [sys.catalog]: This master's current role is: LEADER
I20240503 13:56:10.324726 12955 sys_catalog.cc:458] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130 [sys.catalog]: This master's current role is: LEADER
I20240503 13:56:10.325168 12963 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 13:56:10.328101 12963 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 13:56:10.333751 12963 catalog_manager.cc:1347] Generated new cluster ID: 97784b20396347779e0042367b992316
I20240503 13:56:10.333829 12963 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 13:56:10.344609 12963 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 13:56:10.345731 12963 catalog_manager.cc:1504] Loading token signing keys...
I20240503 13:56:10.354601 12963 catalog_manager.cc:5857] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Generated new TSK 0
I20240503 13:56:10.355345 12963 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 13:56:10.360713 6537 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
/tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.98.65:0
--local_ip_for_outbound_sockets=127.6.98.65
--webserver_interface=127.6.98.65
--webserver_port=0
--tserver_master_addrs=127.6.98.126:44613
--builtin_ntp_servers=127.6.98.84:39449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 13:56:10.487509 12972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:56:10.487717 12972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:56:10.487761 12972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:56:10.490839 12972 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 13:56:10.490917 12972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:56:10.490990 12972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.98.65
I20240503 13:56:10.494578 12972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.98.84:39449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.98.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.6.98.65
--webserver_port=0
--tserver_master_addrs=127.6.98.126:44613
--never_fsync=true
--heap_profile_path=/tmp/kudu.12972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.98.65
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:56:10.495445 12972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:56:10.496584 12972 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:56:10.504998 12977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:56:10.504951 12978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:56:10.505137 12980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:56:10.505399 12972 server_base.cc:1034] running on GCE node
I20240503 13:56:10.505760 12972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:56:10.506281 12972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:56:10.507450 12972 hybrid_clock.cc:648] HybridClock initialized: now 1714744570507435 us; error 34 us; skew 500 ppm
I20240503 13:56:10.509393 12972 webserver.cc:460] Webserver started at http://127.6.98.65:36733/ using document root <none> and password file <none>
I20240503 13:56:10.509929 12972 fs_manager.cc:359] Metadata directory not provided
I20240503 13:56:10.509991 12972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:56:10.510172 12972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:56:10.511811 12972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "60ba3ffe3b7746469834cd91c84be14e"
format_stamp: "Formatted at 2024-05-03 13:56:10 on dist-test-slave-zg9z"
I20240503 13:56:10.512264 12972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "60ba3ffe3b7746469834cd91c84be14e"
format_stamp: "Formatted at 2024-05-03 13:56:10 on dist-test-slave-zg9z"
I20240503 13:56:10.515522 12972 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240503 13:56:10.517618 12986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:56:10.518607 12972 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:10.518718 12972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "60ba3ffe3b7746469834cd91c84be14e"
format_stamp: "Formatted at 2024-05-03 13:56:10 on dist-test-slave-zg9z"
I20240503 13:56:10.518797 12972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:56:10.529808 12972 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:56:10.530401 12972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:56:10.530550 12972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:56:10.531054 12972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:56:10.531888 12972 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:56:10.531932 12972 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:56:10.531971 12972 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:56:10.532009 12972 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:56:10.539863 12972 rpc_server.cc:307] RPC server started. Bound to: 127.6.98.65:34695
I20240503 13:56:10.539901 13099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.98.65:34695 every 8 connection(s)
I20240503 13:56:10.541013 12972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 13:56:10.547230 6537 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu as pid 12972
I20240503 13:56:10.547495 6537 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 13:56:10.549901 6537 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
/tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.98.66:0
--local_ip_for_outbound_sockets=127.6.98.66
--webserver_interface=127.6.98.66
--webserver_port=0
--tserver_master_addrs=127.6.98.126:44613
--builtin_ntp_servers=127.6.98.84:39449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 13:56:10.551050 13100 heartbeater.cc:347] Connected to a master server at 127.6.98.126:44613
I20240503 13:56:10.551281 13100 heartbeater.cc:464] Registering TS with master...
I20240503 13:56:10.551771 13100 heartbeater.cc:511] Master 127.6.98.126:44613 requested a full tablet report, sending...
I20240503 13:56:10.553407 12912 ts_manager.cc:194] Registered new tserver with Master: 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:10.555042 12912 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.98.65:34075
W20240503 13:56:10.656467 13103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:56:10.656697 13103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:56:10.656739 13103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:56:10.659689 13103 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 13:56:10.659752 13103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:56:10.659818 13103 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.98.66
I20240503 13:56:10.663419 13103 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.98.84:39449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.98.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.6.98.66
--webserver_port=0
--tserver_master_addrs=127.6.98.126:44613
--never_fsync=true
--heap_profile_path=/tmp/kudu.13103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.98.66
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:56:10.664249 13103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:56:10.665381 13103 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:56:10.673158 13108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:56:10.673229 13109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:56:10.673302 13111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:56:10.673786 13103 server_base.cc:1034] running on GCE node
I20240503 13:56:10.674124 13103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:56:10.674641 13103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:56:10.675791 13103 hybrid_clock.cc:648] HybridClock initialized: now 1714744570675771 us; error 36 us; skew 500 ppm
I20240503 13:56:10.677906 13103 webserver.cc:460] Webserver started at http://127.6.98.66:42039/ using document root <none> and password file <none>
I20240503 13:56:10.678421 13103 fs_manager.cc:359] Metadata directory not provided
I20240503 13:56:10.678472 13103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:56:10.678629 13103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:56:10.680208 13103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "8ccf5a4ac450414c9d3579bd5800b770"
format_stamp: "Formatted at 2024-05-03 13:56:10 on dist-test-slave-zg9z"
I20240503 13:56:10.680671 13103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "8ccf5a4ac450414c9d3579bd5800b770"
format_stamp: "Formatted at 2024-05-03 13:56:10 on dist-test-slave-zg9z"
I20240503 13:56:10.683799 13103 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20240503 13:56:10.685946 13117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:56:10.686964 13103 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:10.687072 13103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "8ccf5a4ac450414c9d3579bd5800b770"
format_stamp: "Formatted at 2024-05-03 13:56:10 on dist-test-slave-zg9z"
I20240503 13:56:10.687156 13103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:56:10.705890 13103 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:56:10.706491 13103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:56:10.706645 13103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:56:10.707166 13103 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:56:10.708065 13103 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:56:10.708110 13103 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:56:10.708164 13103 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:56:10.708190 13103 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:56:10.716359 13103 rpc_server.cc:307] RPC server started. Bound to: 127.6.98.66:36387
I20240503 13:56:10.716384 13230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.98.66:36387 every 8 connection(s)
I20240503 13:56:10.717271 13103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 13:56:10.726255 6537 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu as pid 13103
I20240503 13:56:10.726384 6537 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 13:56:10.726392 13231 heartbeater.cc:347] Connected to a master server at 127.6.98.126:44613
I20240503 13:56:10.726605 13231 heartbeater.cc:464] Registering TS with master...
I20240503 13:56:10.727102 13231 heartbeater.cc:511] Master 127.6.98.126:44613 requested a full tablet report, sending...
I20240503 13:56:10.728001 12912 ts_manager.cc:194] Registered new tserver with Master: 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:10.728372 6537 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
/tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.98.67:0
--local_ip_for_outbound_sockets=127.6.98.67
--webserver_interface=127.6.98.67
--webserver_port=0
--tserver_master_addrs=127.6.98.126:44613
--builtin_ntp_servers=127.6.98.84:39449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 13:56:10.728615 12912 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.98.66:41393
W20240503 13:56:10.839289 13234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:56:10.839514 13234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:56:10.839563 13234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:56:10.842568 13234 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 13:56:10.842633 13234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:56:10.842700 13234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.98.67
I20240503 13:56:10.846279 13234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.98.84:39449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.98.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.6.98.67
--webserver_port=0
--tserver_master_addrs=127.6.98.126:44613
--never_fsync=true
--heap_profile_path=/tmp/kudu.13234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.98.67
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:56:10.847129 13234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:56:10.848364 13234 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:56:10.856649 13242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:56:10.856715 13239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:56:10.856709 13240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:56:10.857167 13234 server_base.cc:1034] running on GCE node
I20240503 13:56:10.857560 13234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:56:10.858155 13234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:56:10.859328 13234 hybrid_clock.cc:648] HybridClock initialized: now 1714744570859308 us; error 39 us; skew 500 ppm
I20240503 13:56:10.861719 13234 webserver.cc:460] Webserver started at http://127.6.98.67:38973/ using document root <none> and password file <none>
I20240503 13:56:10.862284 13234 fs_manager.cc:359] Metadata directory not provided
I20240503 13:56:10.862339 13234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:56:10.862501 13234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:56:10.864162 13234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "ba0194098aa84449ae17e8a0d4c43703"
format_stamp: "Formatted at 2024-05-03 13:56:10 on dist-test-slave-zg9z"
I20240503 13:56:10.864674 13234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "ba0194098aa84449ae17e8a0d4c43703"
format_stamp: "Formatted at 2024-05-03 13:56:10 on dist-test-slave-zg9z"
I20240503 13:56:10.868070 13234 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240503 13:56:10.870318 13248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:56:10.871354 13234 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240503 13:56:10.871454 13234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "ba0194098aa84449ae17e8a0d4c43703"
format_stamp: "Formatted at 2024-05-03 13:56:10 on dist-test-slave-zg9z"
I20240503 13:56:10.871558 13234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:56:10.895706 13234 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:56:10.896375 13234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:56:10.896561 13234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:56:10.897130 13234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:56:10.898026 13234 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:56:10.898072 13234 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:56:10.898113 13234 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:56:10.898142 13234 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:56:10.906236 13234 rpc_server.cc:307] RPC server started. Bound to: 127.6.98.67:42279
I20240503 13:56:10.906286 13361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.98.67:42279 every 8 connection(s)
I20240503 13:56:10.907167 13234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 13:56:10.914603 6537 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu as pid 13234
I20240503 13:56:10.914709 6537 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 13:56:10.916863 13362 heartbeater.cc:347] Connected to a master server at 127.6.98.126:44613
I20240503 13:56:10.916941 6537 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
/tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.98.68:0
--local_ip_for_outbound_sockets=127.6.98.68
--webserver_interface=127.6.98.68
--webserver_port=0
--tserver_master_addrs=127.6.98.126:44613
--builtin_ntp_servers=127.6.98.84:39449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 13:56:10.917094 13362 heartbeater.cc:464] Registering TS with master...
I20240503 13:56:10.917546 13362 heartbeater.cc:511] Master 127.6.98.126:44613 requested a full tablet report, sending...
I20240503 13:56:10.918522 12912 ts_manager.cc:194] Registered new tserver with Master: ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:10.919108 12912 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.98.67:57473
W20240503 13:56:11.027777 13365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:56:11.028004 13365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:56:11.028052 13365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:56:11.031153 13365 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 13:56:11.031229 13365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:56:11.031309 13365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.98.68
I20240503 13:56:11.035534 13365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.98.84:39449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.98.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.6.98.68
--webserver_port=0
--tserver_master_addrs=127.6.98.126:44613
--never_fsync=true
--heap_profile_path=/tmp/kudu.13365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.98.68
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:56:11.036463 13365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:56:11.037633 13365 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:56:11.045606 13371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:56:11.045701 13370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:56:11.045974 13373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:56:11.046293 13365 server_base.cc:1034] running on GCE node
I20240503 13:56:11.046692 13365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:56:11.047324 13365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:56:11.048534 13365 hybrid_clock.cc:648] HybridClock initialized: now 1714744571048507 us; error 35 us; skew 500 ppm
I20240503 13:56:11.050626 13365 webserver.cc:460] Webserver started at http://127.6.98.68:42803/ using document root <none> and password file <none>
I20240503 13:56:11.051190 13365 fs_manager.cc:359] Metadata directory not provided
I20240503 13:56:11.051245 13365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:56:11.051437 13365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:56:11.053318 13365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
format_stamp: "Formatted at 2024-05-03 13:56:11 on dist-test-slave-zg9z"
I20240503 13:56:11.053970 13365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
format_stamp: "Formatted at 2024-05-03 13:56:11 on dist-test-slave-zg9z"
I20240503 13:56:11.057672 13365 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 13:56:11.060057 13379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:56:11.061125 13365 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 13:56:11.061223 13365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
format_stamp: "Formatted at 2024-05-03 13:56:11 on dist-test-slave-zg9z"
I20240503 13:56:11.061301 13365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:56:11.081015 13365 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:56:11.081625 13365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:56:11.081768 13365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:56:11.082290 13365 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:56:11.083182 13365 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:56:11.083230 13365 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:56:11.083268 13365 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:56:11.083287 13365 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:56:11.092075 13365 rpc_server.cc:307] RPC server started. Bound to: 127.6.98.68:36109
I20240503 13:56:11.092095 13492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.98.68:36109 every 8 connection(s)
I20240503 13:56:11.093060 13365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 13:56:11.093999 6537 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu as pid 13365
I20240503 13:56:11.094097 6537 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 13:56:11.096226 6537 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
/tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.98.69:0
--local_ip_for_outbound_sockets=127.6.98.69
--webserver_interface=127.6.98.69
--webserver_port=0
--tserver_master_addrs=127.6.98.126:44613
--builtin_ntp_servers=127.6.98.84:39449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 13:56:11.102911 13493 heartbeater.cc:347] Connected to a master server at 127.6.98.126:44613
I20240503 13:56:11.103183 13493 heartbeater.cc:464] Registering TS with master...
I20240503 13:56:11.103804 13493 heartbeater.cc:511] Master 127.6.98.126:44613 requested a full tablet report, sending...
I20240503 13:56:11.104969 12912 ts_manager.cc:194] Registered new tserver with Master: 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:11.105723 12912 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.98.68:34299
W20240503 13:56:11.205914 13496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:56:11.206177 13496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:56:11.206238 13496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:56:11.209302 13496 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 13:56:11.209364 13496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:56:11.209432 13496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.98.69
I20240503 13:56:11.213052 13496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.98.84:39449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.98.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.6.98.69
--webserver_port=0
--tserver_master_addrs=127.6.98.126:44613
--never_fsync=true
--heap_profile_path=/tmp/kudu.13496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.98.69
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:56:11.213922 13496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:56:11.215006 13496 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:56:11.223220 13502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:56:11.223477 13501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:56:11.223517 13504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:56:11.223902 13496 server_base.cc:1034] running on GCE node
I20240503 13:56:11.224331 13496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:56:11.224996 13496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:56:11.226150 13496 hybrid_clock.cc:648] HybridClock initialized: now 1714744571226126 us; error 37 us; skew 500 ppm
I20240503 13:56:11.228050 13496 webserver.cc:460] Webserver started at http://127.6.98.69:44007/ using document root <none> and password file <none>
I20240503 13:56:11.228646 13496 fs_manager.cc:359] Metadata directory not provided
I20240503 13:56:11.228705 13496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:56:11.228937 13496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:56:11.230649 13496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "61b887f0d40d497badeca5604c1f0546"
format_stamp: "Formatted at 2024-05-03 13:56:11 on dist-test-slave-zg9z"
I20240503 13:56:11.231117 13496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "61b887f0d40d497badeca5604c1f0546"
format_stamp: "Formatted at 2024-05-03 13:56:11 on dist-test-slave-zg9z"
I20240503 13:56:11.234541 13496 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20240503 13:56:11.236847 13510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:56:11.237891 13496 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:56:11.238023 13496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "61b887f0d40d497badeca5604c1f0546"
format_stamp: "Formatted at 2024-05-03 13:56:11 on dist-test-slave-zg9z"
I20240503 13:56:11.238111 13496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:56:11.251282 13496 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:56:11.251912 13496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:56:11.252084 13496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:56:11.252655 13496 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:56:11.253605 13496 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:56:11.253655 13496 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:56:11.253711 13496 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:56:11.253741 13496 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:56:11.262172 13496 rpc_server.cc:307] RPC server started. Bound to: 127.6.98.69:39153
I20240503 13:56:11.262202 13623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.98.69:39153 every 8 connection(s)
I20240503 13:56:11.263245 13496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 13:56:11.272282 6537 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu as pid 13496
I20240503 13:56:11.272279 13624 heartbeater.cc:347] Connected to a master server at 127.6.98.126:44613
I20240503 13:56:11.272418 6537 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 13:56:11.272540 13624 heartbeater.cc:464] Registering TS with master...
I20240503 13:56:11.272979 13624 heartbeater.cc:511] Master 127.6.98.126:44613 requested a full tablet report, sending...
I20240503 13:56:11.273914 12912 ts_manager.cc:194] Registered new tserver with Master: 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153)
I20240503 13:56:11.274410 6537 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
/tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.98.70:0
--local_ip_for_outbound_sockets=127.6.98.70
--webserver_interface=127.6.98.70
--webserver_port=0
--tserver_master_addrs=127.6.98.126:44613
--builtin_ntp_servers=127.6.98.84:39449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 13:56:11.274438 12912 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.98.69:57517
W20240503 13:56:11.380396 13627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:56:11.380640 13627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:56:11.380699 13627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:56:11.383713 13627 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 13:56:11.383788 13627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:56:11.383874 13627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.98.70
I20240503 13:56:11.387585 13627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.98.84:39449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.98.70:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.6.98.70
--webserver_port=0
--tserver_master_addrs=127.6.98.126:44613
--never_fsync=true
--heap_profile_path=/tmp/kudu.13627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.98.70
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:56:11.388551 13627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:56:11.389683 13627 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:56:11.397774 13632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:56:11.397948 13633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:56:11.398094 13635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:56:11.398448 13627 server_base.cc:1034] running on GCE node
I20240503 13:56:11.398839 13627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:56:11.399463 13627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:56:11.400617 13627 hybrid_clock.cc:648] HybridClock initialized: now 1714744571400607 us; error 33 us; skew 500 ppm
I20240503 13:56:11.402576 13627 webserver.cc:460] Webserver started at http://127.6.98.70:44359/ using document root <none> and password file <none>
I20240503 13:56:11.403123 13627 fs_manager.cc:359] Metadata directory not provided
I20240503 13:56:11.403182 13627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:56:11.403355 13627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:56:11.405066 13627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "009f849ee7b54c80a3f5264695047e4d"
format_stamp: "Formatted at 2024-05-03 13:56:11 on dist-test-slave-zg9z"
I20240503 13:56:11.405524 13627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "009f849ee7b54c80a3f5264695047e4d"
format_stamp: "Formatted at 2024-05-03 13:56:11 on dist-test-slave-zg9z"
I20240503 13:56:11.408916 13627 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 13:56:11.411185 13641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:56:11.412240 13627 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:56:11.412346 13627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "009f849ee7b54c80a3f5264695047e4d"
format_stamp: "Formatted at 2024-05-03 13:56:11 on dist-test-slave-zg9z"
I20240503 13:56:11.412432 13627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:56:11.425140 13627 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:56:11.425752 13627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:56:11.425904 13627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:56:11.426431 13627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:56:11.427299 13627 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:56:11.427346 13627 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:56:11.427398 13627 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:56:11.427423 13627 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:56:11.436232 13627 rpc_server.cc:307] RPC server started. Bound to: 127.6.98.70:34951
I20240503 13:56:11.436254 13754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.98.70:34951 every 8 connection(s)
I20240503 13:56:11.437207 13627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 13:56:11.439869 6537 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu as pid 13627
I20240503 13:56:11.440021 6537 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 13:56:11.442497 6537 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
/tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.98.71:0
--local_ip_for_outbound_sockets=127.6.98.71
--webserver_interface=127.6.98.71
--webserver_port=0
--tserver_master_addrs=127.6.98.126:44613
--builtin_ntp_servers=127.6.98.84:39449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 13:56:11.446507 13755 heartbeater.cc:347] Connected to a master server at 127.6.98.126:44613
I20240503 13:56:11.446719 13755 heartbeater.cc:464] Registering TS with master...
I20240503 13:56:11.447165 13755 heartbeater.cc:511] Master 127.6.98.126:44613 requested a full tablet report, sending...
I20240503 13:56:11.448215 12912 ts_manager.cc:194] Registered new tserver with Master: 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951)
I20240503 13:56:11.449000 12912 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.98.70:39619
W20240503 13:56:11.555150 13758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:56:11.555400 13758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:56:11.555449 13758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 13:56:11.557701 13100 heartbeater.cc:503] Master 127.6.98.126:44613 was elected leader, sending a full tablet report...
W20240503 13:56:11.558805 13758 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 13:56:11.558884 13758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:56:11.558967 13758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.98.71
I20240503 13:56:11.562868 13758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.98.84:39449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.98.71:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.6.98.71
--webserver_port=0
--tserver_master_addrs=127.6.98.126:44613
--never_fsync=true
--heap_profile_path=/tmp/kudu.13758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.98.71
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:56:11.564002 13758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:56:11.565263 13758 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:56:11.573427 13764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:56:11.573546 13766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:56:11.573768 13763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:56:11.574079 13758 server_base.cc:1034] running on GCE node
I20240503 13:56:11.574501 13758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:56:11.575134 13758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:56:11.576301 13758 hybrid_clock.cc:648] HybridClock initialized: now 1714744571576274 us; error 42 us; skew 500 ppm
I20240503 13:56:11.578436 13758 webserver.cc:460] Webserver started at http://127.6.98.71:45699/ using document root <none> and password file <none>
I20240503 13:56:11.579003 13758 fs_manager.cc:359] Metadata directory not provided
I20240503 13:56:11.579063 13758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:56:11.579239 13758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:56:11.580965 13758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "76c386bef61a4bd5a907266a5118ffe2"
format_stamp: "Formatted at 2024-05-03 13:56:11 on dist-test-slave-zg9z"
I20240503 13:56:11.581431 13758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "76c386bef61a4bd5a907266a5118ffe2"
format_stamp: "Formatted at 2024-05-03 13:56:11 on dist-test-slave-zg9z"
I20240503 13:56:11.584861 13758 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 13:56:11.587121 13772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:56:11.588526 13758 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:56:11.588649 13758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "76c386bef61a4bd5a907266a5118ffe2"
format_stamp: "Formatted at 2024-05-03 13:56:11 on dist-test-slave-zg9z"
I20240503 13:56:11.588742 13758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:56:11.607887 13758 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:56:11.608563 13758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:56:11.608712 13758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:56:11.609243 13758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:56:11.610309 13758 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:56:11.610365 13758 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:56:11.610409 13758 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:56:11.610421 13758 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:56:11.619570 13758 rpc_server.cc:307] RPC server started. Bound to: 127.6.98.71:42885
I20240503 13:56:11.619655 13885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.98.71:42885 every 8 connection(s)
I20240503 13:56:11.620754 13758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 13:56:11.628722 6537 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu as pid 13758
I20240503 13:56:11.628839 6537 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 13:56:11.630904 13886 heartbeater.cc:347] Connected to a master server at 127.6.98.126:44613
I20240503 13:56:11.630968 6537 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
/tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.98.72:0
--local_ip_for_outbound_sockets=127.6.98.72
--webserver_interface=127.6.98.72
--webserver_port=0
--tserver_master_addrs=127.6.98.126:44613
--builtin_ntp_servers=127.6.98.84:39449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 13:56:11.631178 13886 heartbeater.cc:464] Registering TS with master...
I20240503 13:56:11.631747 13886 heartbeater.cc:511] Master 127.6.98.126:44613 requested a full tablet report, sending...
I20240503 13:56:11.632803 12912 ts_manager.cc:194] Registered new tserver with Master: 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885)
I20240503 13:56:11.633360 12912 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.98.71:53373
I20240503 13:56:11.730721 13231 heartbeater.cc:503] Master 127.6.98.126:44613 was elected leader, sending a full tablet report...
W20240503 13:56:11.745867 13889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:56:11.746104 13889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:56:11.746150 13889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:56:11.749261 13889 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 13:56:11.749343 13889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:56:11.749419 13889 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.98.72
I20240503 13:56:11.753255 13889 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.98.84:39449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.98.72:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.6.98.72
--webserver_port=0
--tserver_master_addrs=127.6.98.126:44613
--never_fsync=true
--heap_profile_path=/tmp/kudu.13889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.98.72
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:56:11.754233 13889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:56:11.755385 13889 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:56:11.764654 13897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:56:11.764654 13894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:56:11.765241 13895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:56:11.765429 13889 server_base.cc:1034] running on GCE node
I20240503 13:56:11.765903 13889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:56:11.766698 13889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:56:11.767885 13889 hybrid_clock.cc:648] HybridClock initialized: now 1714744571767860 us; error 43 us; skew 500 ppm
I20240503 13:56:11.770253 13889 webserver.cc:460] Webserver started at http://127.6.98.72:45043/ using document root <none> and password file <none>
I20240503 13:56:11.770972 13889 fs_manager.cc:359] Metadata directory not provided
I20240503 13:56:11.771042 13889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:56:11.771225 13889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:56:11.773223 13889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "e47105a0f46e4511b7112e571688a4c7"
format_stamp: "Formatted at 2024-05-03 13:56:11 on dist-test-slave-zg9z"
I20240503 13:56:11.773751 13889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "e47105a0f46e4511b7112e571688a4c7"
format_stamp: "Formatted at 2024-05-03 13:56:11 on dist-test-slave-zg9z"
I20240503 13:56:11.777524 13889 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240503 13:56:11.779850 13903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:56:11.781030 13889 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:56:11.781155 13889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "e47105a0f46e4511b7112e571688a4c7"
format_stamp: "Formatted at 2024-05-03 13:56:11 on dist-test-slave-zg9z"
I20240503 13:56:11.781256 13889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:56:11.802829 13889 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:56:11.803496 13889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:56:11.803655 13889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:56:11.804180 13889 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:56:11.805179 13889 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:56:11.805230 13889 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:56:11.805269 13889 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:56:11.805285 13889 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:56:11.813501 13889 rpc_server.cc:307] RPC server started. Bound to: 127.6.98.72:44753
I20240503 13:56:11.813531 14016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.98.72:44753 every 8 connection(s)
I20240503 13:56:11.814445 13889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 13:56:11.817699 6537 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu as pid 13889
I20240503 13:56:11.817785 6537 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 13:56:11.823408 14017 heartbeater.cc:347] Connected to a master server at 127.6.98.126:44613
I20240503 13:56:11.823647 14017 heartbeater.cc:464] Registering TS with master...
I20240503 13:56:11.824095 14017 heartbeater.cc:511] Master 127.6.98.126:44613 requested a full tablet report, sending...
I20240503 13:56:11.825152 12912 ts_manager.cc:194] Registered new tserver with Master: e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:11.825711 12912 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.98.72:34463
I20240503 13:56:11.832206 6537 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 13:56:11.842005 6537 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 13:56:11.842095 6537 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu with pid 13496
I20240503 13:56:11.846941 6537 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu with pid 13627
I20240503 13:56:11.851632 6537 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu with pid 13758
I20240503 13:56:11.855989 6537 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu with pid 13889
I20240503 13:56:11.921701 13362 heartbeater.cc:503] Master 127.6.98.126:44613 was elected leader, sending a full tablet report...
I20240503 13:56:12.108081 13493 heartbeater.cc:503] Master 127.6.98.126:44613 was elected leader, sending a full tablet report...
I20240503 13:56:15.612458 12912 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:56:15.634721 13034 tablet_service.cc:1467] Processing CreateTablet for tablet 101417485ac24fd9bb868fd81b2552e5 (DEFAULT_TABLE table=rebalance_test_table_0 [id=81d7faaf4b654c90b42092653455de5d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:15.634721 13165 tablet_service.cc:1467] Processing CreateTablet for tablet 5f79adb3e75c4a45998c73f29b65e312 (DEFAULT_TABLE table=rebalance_test_table_0 [id=81d7faaf4b654c90b42092653455de5d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:15.635761 13034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 101417485ac24fd9bb868fd81b2552e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:15.635761 13165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f79adb3e75c4a45998c73f29b65e312. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:15.635416 13427 tablet_service.cc:1467] Processing CreateTablet for tablet 101417485ac24fd9bb868fd81b2552e5 (DEFAULT_TABLE table=rebalance_test_table_0 [id=81d7faaf4b654c90b42092653455de5d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:15.636132 13032 tablet_service.cc:1467] Processing CreateTablet for tablet 3dd1e318c2084e3a9a80987731920ac8 (DEFAULT_TABLE table=rebalance_test_table_0 [id=81d7faaf4b654c90b42092653455de5d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:15.636282 13427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 101417485ac24fd9bb868fd81b2552e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:15.636596 13032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3dd1e318c2084e3a9a80987731920ac8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:15.636744 13033 tablet_service.cc:1467] Processing CreateTablet for tablet 5f79adb3e75c4a45998c73f29b65e312 (DEFAULT_TABLE table=rebalance_test_table_0 [id=81d7faaf4b654c90b42092653455de5d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:15.636966 13033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f79adb3e75c4a45998c73f29b65e312. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:15.638770 13164 tablet_service.cc:1467] Processing CreateTablet for tablet 3dd1e318c2084e3a9a80987731920ac8 (DEFAULT_TABLE table=rebalance_test_table_0 [id=81d7faaf4b654c90b42092653455de5d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:15.639027 13164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3dd1e318c2084e3a9a80987731920ac8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:15.635416 13426 tablet_service.cc:1467] Processing CreateTablet for tablet 3dd1e318c2084e3a9a80987731920ac8 (DEFAULT_TABLE table=rebalance_test_table_0 [id=81d7faaf4b654c90b42092653455de5d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:15.640698 13426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3dd1e318c2084e3a9a80987731920ac8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:15.643695 14038 tablet_bootstrap.cc:492] T 3dd1e318c2084e3a9a80987731920ac8 P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:15.643755 14039 tablet_bootstrap.cc:492] T 3dd1e318c2084e3a9a80987731920ac8 P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:15.645988 14038 tablet_bootstrap.cc:654] T 3dd1e318c2084e3a9a80987731920ac8 P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:15.646322 14039 tablet_bootstrap.cc:654] T 3dd1e318c2084e3a9a80987731920ac8 P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:15.646889 14038 log.cc:824] T 3dd1e318c2084e3a9a80987731920ac8 P 7690ee79577c45e3a2c8f8b552c0bff8: Log is configured to *not* fsync() on all Append() calls
I20240503 13:56:15.647161 14039 log.cc:824] T 3dd1e318c2084e3a9a80987731920ac8 P 8ccf5a4ac450414c9d3579bd5800b770: Log is configured to *not* fsync() on all Append() calls
I20240503 13:56:15.649325 14037 tablet_bootstrap.cc:492] T 101417485ac24fd9bb868fd81b2552e5 P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:15.649349 13296 tablet_service.cc:1467] Processing CreateTablet for tablet 101417485ac24fd9bb868fd81b2552e5 (DEFAULT_TABLE table=rebalance_test_table_0 [id=81d7faaf4b654c90b42092653455de5d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:15.650173 14039 tablet_bootstrap.cc:492] T 3dd1e318c2084e3a9a80987731920ac8 P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:15.650341 13296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 101417485ac24fd9bb868fd81b2552e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:15.650354 14039 ts_tablet_manager.cc:1367] T 3dd1e318c2084e3a9a80987731920ac8 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240503 13:56:15.651091 14037 tablet_bootstrap.cc:654] T 101417485ac24fd9bb868fd81b2552e5 P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:15.651657 14037 log.cc:824] T 101417485ac24fd9bb868fd81b2552e5 P 60ba3ffe3b7746469834cd91c84be14e: Log is configured to *not* fsync() on all Append() calls
I20240503 13:56:15.652803 13295 tablet_service.cc:1467] Processing CreateTablet for tablet 5f79adb3e75c4a45998c73f29b65e312 (DEFAULT_TABLE table=rebalance_test_table_0 [id=81d7faaf4b654c90b42092653455de5d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:15.653071 13295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f79adb3e75c4a45998c73f29b65e312. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:15.653223 14038 tablet_bootstrap.cc:492] T 3dd1e318c2084e3a9a80987731920ac8 P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:15.653582 14038 ts_tablet_manager.cc:1367] T 3dd1e318c2084e3a9a80987731920ac8 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.004s
I20240503 13:56:15.654232 14037 tablet_bootstrap.cc:492] T 101417485ac24fd9bb868fd81b2552e5 P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:15.654448 14037 ts_tablet_manager.cc:1367] T 101417485ac24fd9bb868fd81b2552e5 P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 13:56:15.654888 14039 raft_consensus.cc:348] T 3dd1e318c2084e3a9a80987731920ac8 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:15.655105 14039 raft_consensus.cc:374] T 3dd1e318c2084e3a9a80987731920ac8 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:15.655157 14039 raft_consensus.cc:729] T 3dd1e318c2084e3a9a80987731920ac8 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:15.655630 14039 consensus_queue.cc:260] T 3dd1e318c2084e3a9a80987731920ac8 P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:15.656535 14039 ts_tablet_manager.cc:1398] T 3dd1e318c2084e3a9a80987731920ac8 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20240503 13:56:15.656901 14039 tablet_bootstrap.cc:492] T 5f79adb3e75c4a45998c73f29b65e312 P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:15.658110 14037 raft_consensus.cc:348] T 101417485ac24fd9bb868fd81b2552e5 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:15.658370 14037 raft_consensus.cc:374] T 101417485ac24fd9bb868fd81b2552e5 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:15.658460 14037 raft_consensus.cc:729] T 101417485ac24fd9bb868fd81b2552e5 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:15.658375 14038 raft_consensus.cc:348] T 3dd1e318c2084e3a9a80987731920ac8 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:15.658555 14038 raft_consensus.cc:374] T 3dd1e318c2084e3a9a80987731920ac8 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:15.658607 14038 raft_consensus.cc:729] T 3dd1e318c2084e3a9a80987731920ac8 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:15.659035 14038 consensus_queue.cc:260] T 3dd1e318c2084e3a9a80987731920ac8 P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:15.659035 14037 consensus_queue.cc:260] T 101417485ac24fd9bb868fd81b2552e5 P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:15.659883 14038 ts_tablet_manager.cc:1398] T 3dd1e318c2084e3a9a80987731920ac8 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240503 13:56:15.659883 14037 ts_tablet_manager.cc:1398] T 101417485ac24fd9bb868fd81b2552e5 P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240503 13:56:15.660246 14038 tablet_bootstrap.cc:492] T 101417485ac24fd9bb868fd81b2552e5 P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:15.660670 14039 tablet_bootstrap.cc:654] T 5f79adb3e75c4a45998c73f29b65e312 P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:15.661192 14038 tablet_bootstrap.cc:654] T 101417485ac24fd9bb868fd81b2552e5 P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:15.661486 14044 tablet_bootstrap.cc:492] T 5f79adb3e75c4a45998c73f29b65e312 P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:15.662146 14037 tablet_bootstrap.cc:492] T 3dd1e318c2084e3a9a80987731920ac8 P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:15.662292 14038 tablet_bootstrap.cc:492] T 101417485ac24fd9bb868fd81b2552e5 P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:15.662374 14038 ts_tablet_manager.cc:1367] T 101417485ac24fd9bb868fd81b2552e5 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:15.662904 14038 raft_consensus.cc:348] T 101417485ac24fd9bb868fd81b2552e5 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:15.663023 14038 raft_consensus.cc:374] T 101417485ac24fd9bb868fd81b2552e5 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:15.663062 14038 raft_consensus.cc:729] T 101417485ac24fd9bb868fd81b2552e5 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:15.663211 14038 consensus_queue.cc:260] T 101417485ac24fd9bb868fd81b2552e5 P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:15.663573 14038 ts_tablet_manager.cc:1398] T 101417485ac24fd9bb868fd81b2552e5 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:15.663646 14044 tablet_bootstrap.cc:654] T 5f79adb3e75c4a45998c73f29b65e312 P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:15.664037 14037 tablet_bootstrap.cc:654] T 3dd1e318c2084e3a9a80987731920ac8 P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:15.664449 14039 tablet_bootstrap.cc:492] T 5f79adb3e75c4a45998c73f29b65e312 P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:15.664436 14044 log.cc:824] T 5f79adb3e75c4a45998c73f29b65e312 P ba0194098aa84449ae17e8a0d4c43703: Log is configured to *not* fsync() on all Append() calls
I20240503 13:56:15.664558 14039 ts_tablet_manager.cc:1367] T 5f79adb3e75c4a45998c73f29b65e312 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240503 13:56:15.665130 14039 raft_consensus.cc:348] T 5f79adb3e75c4a45998c73f29b65e312 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:15.665263 14039 raft_consensus.cc:374] T 5f79adb3e75c4a45998c73f29b65e312 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:15.665307 14039 raft_consensus.cc:729] T 5f79adb3e75c4a45998c73f29b65e312 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:15.665345 14037 tablet_bootstrap.cc:492] T 3dd1e318c2084e3a9a80987731920ac8 P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:15.665438 14037 ts_tablet_manager.cc:1367] T 3dd1e318c2084e3a9a80987731920ac8 P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 13:56:15.665436 14039 consensus_queue.cc:260] T 5f79adb3e75c4a45998c73f29b65e312 P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:15.665694 14039 ts_tablet_manager.cc:1398] T 5f79adb3e75c4a45998c73f29b65e312 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:15.665931 14037 raft_consensus.cc:348] T 3dd1e318c2084e3a9a80987731920ac8 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:15.666060 14037 raft_consensus.cc:374] T 3dd1e318c2084e3a9a80987731920ac8 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:15.666100 14037 raft_consensus.cc:729] T 3dd1e318c2084e3a9a80987731920ac8 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:15.666217 14037 consensus_queue.cc:260] T 3dd1e318c2084e3a9a80987731920ac8 P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:15.666499 14037 ts_tablet_manager.cc:1398] T 3dd1e318c2084e3a9a80987731920ac8 P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:15.666672 14037 tablet_bootstrap.cc:492] T 5f79adb3e75c4a45998c73f29b65e312 P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:15.667526 14037 tablet_bootstrap.cc:654] T 5f79adb3e75c4a45998c73f29b65e312 P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:15.668718 14037 tablet_bootstrap.cc:492] T 5f79adb3e75c4a45998c73f29b65e312 P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:15.668810 14037 ts_tablet_manager.cc:1367] T 5f79adb3e75c4a45998c73f29b65e312 P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 13:56:15.669196 14037 raft_consensus.cc:348] T 5f79adb3e75c4a45998c73f29b65e312 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:15.669281 14037 raft_consensus.cc:374] T 5f79adb3e75c4a45998c73f29b65e312 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:15.669308 14037 raft_consensus.cc:729] T 5f79adb3e75c4a45998c73f29b65e312 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:15.669413 14037 consensus_queue.cc:260] T 5f79adb3e75c4a45998c73f29b65e312 P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:15.669674 14037 ts_tablet_manager.cc:1398] T 5f79adb3e75c4a45998c73f29b65e312 P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:15.670050 14044 tablet_bootstrap.cc:492] T 5f79adb3e75c4a45998c73f29b65e312 P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:15.670240 14044 ts_tablet_manager.cc:1367] T 5f79adb3e75c4a45998c73f29b65e312 P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240503 13:56:15.673614 14044 raft_consensus.cc:348] T 5f79adb3e75c4a45998c73f29b65e312 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:15.673794 14044 raft_consensus.cc:374] T 5f79adb3e75c4a45998c73f29b65e312 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:15.673851 14044 raft_consensus.cc:729] T 5f79adb3e75c4a45998c73f29b65e312 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:15.674268 14044 consensus_queue.cc:260] T 5f79adb3e75c4a45998c73f29b65e312 P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:15.675143 14044 ts_tablet_manager.cc:1398] T 5f79adb3e75c4a45998c73f29b65e312 P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240503 13:56:15.675474 14044 tablet_bootstrap.cc:492] T 101417485ac24fd9bb868fd81b2552e5 P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:15.676463 14044 tablet_bootstrap.cc:654] T 101417485ac24fd9bb868fd81b2552e5 P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:15.677568 14044 tablet_bootstrap.cc:492] T 101417485ac24fd9bb868fd81b2552e5 P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:15.677637 14044 ts_tablet_manager.cc:1367] T 101417485ac24fd9bb868fd81b2552e5 P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:56:15.678066 14044 raft_consensus.cc:348] T 101417485ac24fd9bb868fd81b2552e5 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:15.678160 14044 raft_consensus.cc:374] T 101417485ac24fd9bb868fd81b2552e5 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:15.678190 14044 raft_consensus.cc:729] T 101417485ac24fd9bb868fd81b2552e5 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:15.678300 14044 consensus_queue.cc:260] T 101417485ac24fd9bb868fd81b2552e5 P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:15.678551 14044 ts_tablet_manager.cc:1398] T 101417485ac24fd9bb868fd81b2552e5 P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20240503 13:56:15.722057 13232 tablet.cc:2369] T 3dd1e318c2084e3a9a80987731920ac8 P 8ccf5a4ac450414c9d3579bd5800b770: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 13:56:15.754365 14046 raft_consensus.cc:482] T 5f79adb3e75c4a45998c73f29b65e312 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:15.754565 14046 raft_consensus.cc:504] T 5f79adb3e75c4a45998c73f29b65e312 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:15.755671 14046 leader_election.cc:288] T 5f79adb3e75c4a45998c73f29b65e312 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387), ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:15.760385 13185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f79adb3e75c4a45998c73f29b65e312" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770" is_pre_election: true
I20240503 13:56:15.760579 13316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f79adb3e75c4a45998c73f29b65e312" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703" is_pre_election: true
I20240503 13:56:15.760753 13185 raft_consensus.cc:2454] T 5f79adb3e75c4a45998c73f29b65e312 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 0.
I20240503 13:56:15.760809 13316 raft_consensus.cc:2454] T 5f79adb3e75c4a45998c73f29b65e312 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 0.
I20240503 13:56:15.761201 12990 leader_election.cc:302] T 5f79adb3e75c4a45998c73f29b65e312 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:15.761452 14046 raft_consensus.cc:2781] T 5f79adb3e75c4a45998c73f29b65e312 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:15.761528 14046 raft_consensus.cc:482] T 5f79adb3e75c4a45998c73f29b65e312 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:15.761577 14046 raft_consensus.cc:3037] T 5f79adb3e75c4a45998c73f29b65e312 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:15.762450 14046 raft_consensus.cc:504] T 5f79adb3e75c4a45998c73f29b65e312 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:15.762781 14046 leader_election.cc:288] T 5f79adb3e75c4a45998c73f29b65e312 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 election: Requested vote from peers 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387), ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:15.763244 13185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f79adb3e75c4a45998c73f29b65e312" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770"
I20240503 13:56:15.763248 13316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f79adb3e75c4a45998c73f29b65e312" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703"
I20240503 13:56:15.763382 13185 raft_consensus.cc:3037] T 5f79adb3e75c4a45998c73f29b65e312 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:15.763401 13316 raft_consensus.cc:3037] T 5f79adb3e75c4a45998c73f29b65e312 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:15.764515 13185 raft_consensus.cc:2454] T 5f79adb3e75c4a45998c73f29b65e312 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 1.
I20240503 13:56:15.764515 13316 raft_consensus.cc:2454] T 5f79adb3e75c4a45998c73f29b65e312 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 1.
I20240503 13:56:15.764812 12990 leader_election.cc:302] T 5f79adb3e75c4a45998c73f29b65e312 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:15.765027 14046 raft_consensus.cc:2781] T 5f79adb3e75c4a45998c73f29b65e312 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:15.765257 14046 raft_consensus.cc:686] T 5f79adb3e75c4a45998c73f29b65e312 P 60ba3ffe3b7746469834cd91c84be14e [term 1 LEADER]: Becoming Leader. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Running, Role: LEADER
I20240503 13:56:15.765610 14046 consensus_queue.cc:237] T 5f79adb3e75c4a45998c73f29b65e312 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:15.768512 12912 catalog_manager.cc:5484] T 5f79adb3e75c4a45998c73f29b65e312 P 60ba3ffe3b7746469834cd91c84be14e reported cstate change: term changed from 0 to 1, leader changed from <none> to 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65). New cstate: current_term: 1 leader_uuid: "60ba3ffe3b7746469834cd91c84be14e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
W20240503 13:56:15.796171 13101 tablet.cc:2369] T 101417485ac24fd9bb868fd81b2552e5 P 60ba3ffe3b7746469834cd91c84be14e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 13:56:15.828163 14049 raft_consensus.cc:482] T 101417485ac24fd9bb868fd81b2552e5 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:15.828359 14049 raft_consensus.cc:504] T 101417485ac24fd9bb868fd81b2552e5 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:15.829712 14049 leader_election.cc:288] T 101417485ac24fd9bb868fd81b2552e5 P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279), 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:15.833631 13054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "101417485ac24fd9bb868fd81b2552e5" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e" is_pre_election: true
I20240503 13:56:15.833628 13316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "101417485ac24fd9bb868fd81b2552e5" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703" is_pre_election: true
I20240503 13:56:15.833814 13316 raft_consensus.cc:2454] T 101417485ac24fd9bb868fd81b2552e5 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 0.
I20240503 13:56:15.833858 13054 raft_consensus.cc:2454] T 101417485ac24fd9bb868fd81b2552e5 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 0.
I20240503 13:56:15.834201 13383 leader_election.cc:302] T 101417485ac24fd9bb868fd81b2552e5 P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:15.834482 14049 raft_consensus.cc:2781] T 101417485ac24fd9bb868fd81b2552e5 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:15.834559 14049 raft_consensus.cc:482] T 101417485ac24fd9bb868fd81b2552e5 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:15.834599 14049 raft_consensus.cc:3037] T 101417485ac24fd9bb868fd81b2552e5 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:15.835518 14049 raft_consensus.cc:504] T 101417485ac24fd9bb868fd81b2552e5 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:15.835862 14049 leader_election.cc:288] T 101417485ac24fd9bb868fd81b2552e5 P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 election: Requested vote from peers ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279), 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:15.836222 13316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "101417485ac24fd9bb868fd81b2552e5" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703"
I20240503 13:56:15.836359 13316 raft_consensus.cc:3037] T 101417485ac24fd9bb868fd81b2552e5 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:15.836342 13054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "101417485ac24fd9bb868fd81b2552e5" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e"
I20240503 13:56:15.836506 13054 raft_consensus.cc:3037] T 101417485ac24fd9bb868fd81b2552e5 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:15.837430 13316 raft_consensus.cc:2454] T 101417485ac24fd9bb868fd81b2552e5 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 1.
I20240503 13:56:15.837466 13054 raft_consensus.cc:2454] T 101417485ac24fd9bb868fd81b2552e5 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 1.
I20240503 13:56:15.837749 13383 leader_election.cc:302] T 101417485ac24fd9bb868fd81b2552e5 P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:15.837924 14049 raft_consensus.cc:2781] T 101417485ac24fd9bb868fd81b2552e5 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:15.838027 14049 raft_consensus.cc:686] T 101417485ac24fd9bb868fd81b2552e5 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 LEADER]: Becoming Leader. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Running, Role: LEADER
I20240503 13:56:15.838316 14049 consensus_queue.cc:237] T 101417485ac24fd9bb868fd81b2552e5 P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:15.841219 12912 catalog_manager.cc:5484] T 101417485ac24fd9bb868fd81b2552e5 P 7690ee79577c45e3a2c8f8b552c0bff8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68). New cstate: current_term: 1 leader_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } }
W20240503 13:56:15.847715 13494 tablet.cc:2369] T 101417485ac24fd9bb868fd81b2552e5 P 7690ee79577c45e3a2c8f8b552c0bff8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 13:56:15.912048 13363 tablet.cc:2369] T 101417485ac24fd9bb868fd81b2552e5 P ba0194098aa84449ae17e8a0d4c43703: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 13:56:15.995720 14049 raft_consensus.cc:482] T 3dd1e318c2084e3a9a80987731920ac8 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:15.995867 14049 raft_consensus.cc:504] T 3dd1e318c2084e3a9a80987731920ac8 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:15.996379 14049 leader_election.cc:288] T 3dd1e318c2084e3a9a80987731920ac8 P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:15.996995 13054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dd1e318c2084e3a9a80987731920ac8" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e" is_pre_election: true
I20240503 13:56:15.997135 13054 raft_consensus.cc:2454] T 3dd1e318c2084e3a9a80987731920ac8 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 0.
I20240503 13:56:15.997447 13381 leader_election.cc:302] T 3dd1e318c2084e3a9a80987731920ac8 P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 7690ee79577c45e3a2c8f8b552c0bff8; no voters:
I20240503 13:56:15.997608 14049 raft_consensus.cc:2781] T 3dd1e318c2084e3a9a80987731920ac8 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:15.997678 14049 raft_consensus.cc:482] T 3dd1e318c2084e3a9a80987731920ac8 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:15.997712 14049 raft_consensus.cc:3037] T 3dd1e318c2084e3a9a80987731920ac8 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:15.998729 14049 raft_consensus.cc:504] T 3dd1e318c2084e3a9a80987731920ac8 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:15.999089 14049 leader_election.cc:288] T 3dd1e318c2084e3a9a80987731920ac8 P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 election: Requested vote from peers 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:15.999402 13054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dd1e318c2084e3a9a80987731920ac8" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e"
I20240503 13:56:15.999533 13054 raft_consensus.cc:3037] T 3dd1e318c2084e3a9a80987731920ac8 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:16.000245 13185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dd1e318c2084e3a9a80987731920ac8" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770" is_pre_election: true
I20240503 13:56:16.000278 13184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dd1e318c2084e3a9a80987731920ac8" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770"
I20240503 13:56:16.000360 13054 raft_consensus.cc:2454] T 3dd1e318c2084e3a9a80987731920ac8 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 1.
I20240503 13:56:16.000402 13185 raft_consensus.cc:2454] T 3dd1e318c2084e3a9a80987731920ac8 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 0.
I20240503 13:56:16.000634 13381 leader_election.cc:302] T 3dd1e318c2084e3a9a80987731920ac8 P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 7690ee79577c45e3a2c8f8b552c0bff8; no voters:
I20240503 13:56:16.000826 14049 raft_consensus.cc:2781] T 3dd1e318c2084e3a9a80987731920ac8 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:16.000909 14049 raft_consensus.cc:686] T 3dd1e318c2084e3a9a80987731920ac8 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 LEADER]: Becoming Leader. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Running, Role: LEADER
I20240503 13:56:16.001067 14049 consensus_queue.cc:237] T 3dd1e318c2084e3a9a80987731920ac8 P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:16.002604 12912 catalog_manager.cc:5484] T 3dd1e318c2084e3a9a80987731920ac8 P 7690ee79577c45e3a2c8f8b552c0bff8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68). New cstate: current_term: 1 leader_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } }
I20240503 13:56:16.143680 14046 consensus_queue.cc:1035] T 5f79adb3e75c4a45998c73f29b65e312 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:16.149219 14057 consensus_queue.cc:1035] T 5f79adb3e75c4a45998c73f29b65e312 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 13:56:16.197675 14063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:56:16.197932 14063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:56:16.201004 14063 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240503 13:56:16.294684 13054 raft_consensus.cc:1261] T 101417485ac24fd9bb868fd81b2552e5 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Refusing update from remote peer 7690ee79577c45e3a2c8f8b552c0bff8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:16.295344 14049 consensus_queue.cc:1035] T 101417485ac24fd9bb868fd81b2552e5 P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:16.296331 13184 raft_consensus.cc:3037] T 3dd1e318c2084e3a9a80987731920ac8 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:16.297528 13184 raft_consensus.cc:1261] T 3dd1e318c2084e3a9a80987731920ac8 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Refusing update from remote peer 7690ee79577c45e3a2c8f8b552c0bff8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:16.297997 14045 consensus_queue.cc:1035] T 3dd1e318c2084e3a9a80987731920ac8 P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:16.298506 13053 raft_consensus.cc:1261] T 3dd1e318c2084e3a9a80987731920ac8 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Refusing update from remote peer 7690ee79577c45e3a2c8f8b552c0bff8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:16.300614 14045 consensus_queue.cc:1035] T 3dd1e318c2084e3a9a80987731920ac8 P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:16.308038 14099 mvcc.cc:205] Tried to move back new op lower bound from 7023593784496537600 to 7023593782636175360. Current Snapshot: MvccSnapshot[applied={T|T < 7023593784496537600}]
I20240503 13:56:16.308241 14104 mvcc.cc:205] Tried to move back new op lower bound from 7023593784496537600 to 7023593782636175360. Current Snapshot: MvccSnapshot[applied={T|T < 7023593784496537600}]
I20240503 13:56:16.308254 13315 raft_consensus.cc:1261] T 101417485ac24fd9bb868fd81b2552e5 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Refusing update from remote peer 7690ee79577c45e3a2c8f8b552c0bff8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:16.308717 14045 consensus_queue.cc:1035] T 101417485ac24fd9bb868fd81b2552e5 P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:16.332718 14111 mvcc.cc:205] Tried to move back new op lower bound from 7023593784512700416 to 7023593782636175360. Current Snapshot: MvccSnapshot[applied={T|T < 7023593784496537600}]
I20240503 13:56:16.346119 14066 mvcc.cc:205] Tried to move back new op lower bound from 7023593784505442304 to 7023593783301640192. Current Snapshot: MvccSnapshot[applied={T|T < 7023593784505442304}]
INSERT report
rows total: 2000
time total: 108.103 ms
time per row: 0.0540516 ms
I20240503 13:56:16.360718 12906 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:56:16.366482 13427 tablet_service.cc:1467] Processing CreateTablet for tablet 7239f94a7c88433aa6062390a54e7562 (DEFAULT_TABLE table=rebalance_test_table_1 [id=ba90ff8253204f25829d57a56c0b5e94]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:16.366473 13165 tablet_service.cc:1467] Processing CreateTablet for tablet 7239f94a7c88433aa6062390a54e7562 (DEFAULT_TABLE table=rebalance_test_table_1 [id=ba90ff8253204f25829d57a56c0b5e94]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:16.366596 13296 tablet_service.cc:1467] Processing CreateTablet for tablet 7239f94a7c88433aa6062390a54e7562 (DEFAULT_TABLE table=rebalance_test_table_1 [id=ba90ff8253204f25829d57a56c0b5e94]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:16.366791 13427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7239f94a7c88433aa6062390a54e7562. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:16.366791 13165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7239f94a7c88433aa6062390a54e7562. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:16.366899 13296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7239f94a7c88433aa6062390a54e7562. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:16.367179 13426 tablet_service.cc:1467] Processing CreateTablet for tablet 477c33f203f649019eff2b28524fad90 (DEFAULT_TABLE table=rebalance_test_table_1 [id=ba90ff8253204f25829d57a56c0b5e94]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:16.367384 13426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 477c33f203f649019eff2b28524fad90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:16.369226 14125 tablet_bootstrap.cc:492] T 7239f94a7c88433aa6062390a54e7562 P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:16.369680 14126 tablet_bootstrap.cc:492] T 7239f94a7c88433aa6062390a54e7562 P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:16.369771 13033 tablet_service.cc:1467] Processing CreateTablet for tablet df28831df92d4c12b63b2f7da6c6449a (DEFAULT_TABLE table=rebalance_test_table_1 [id=ba90ff8253204f25829d57a56c0b5e94]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:16.370036 13033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df28831df92d4c12b63b2f7da6c6449a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:16.370402 14125 tablet_bootstrap.cc:654] T 7239f94a7c88433aa6062390a54e7562 P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:16.370528 14126 tablet_bootstrap.cc:654] T 7239f94a7c88433aa6062390a54e7562 P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:16.371476 14126 tablet_bootstrap.cc:492] T 7239f94a7c88433aa6062390a54e7562 P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:16.371537 14126 ts_tablet_manager.cc:1367] T 7239f94a7c88433aa6062390a54e7562 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:16.371721 14125 tablet_bootstrap.cc:492] T 7239f94a7c88433aa6062390a54e7562 P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:16.371795 14125 ts_tablet_manager.cc:1367] T 7239f94a7c88433aa6062390a54e7562 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:56:16.371910 14126 raft_consensus.cc:348] T 7239f94a7c88433aa6062390a54e7562 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:16.372025 14126 raft_consensus.cc:374] T 7239f94a7c88433aa6062390a54e7562 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:16.372058 14126 raft_consensus.cc:729] T 7239f94a7c88433aa6062390a54e7562 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:16.372179 14126 consensus_queue.cc:260] T 7239f94a7c88433aa6062390a54e7562 P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:16.372205 14125 raft_consensus.cc:348] T 7239f94a7c88433aa6062390a54e7562 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:16.372411 14125 raft_consensus.cc:374] T 7239f94a7c88433aa6062390a54e7562 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:16.372447 14125 raft_consensus.cc:729] T 7239f94a7c88433aa6062390a54e7562 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:16.372608 14125 consensus_queue.cc:260] T 7239f94a7c88433aa6062390a54e7562 P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:16.373155 13165 tablet_service.cc:1467] Processing CreateTablet for tablet 477c33f203f649019eff2b28524fad90 (DEFAULT_TABLE table=rebalance_test_table_1 [id=ba90ff8253204f25829d57a56c0b5e94]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:16.373369 13165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 477c33f203f649019eff2b28524fad90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:16.373339 13296 tablet_service.cc:1467] Processing CreateTablet for tablet 477c33f203f649019eff2b28524fad90 (DEFAULT_TABLE table=rebalance_test_table_1 [id=ba90ff8253204f25829d57a56c0b5e94]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:16.373373 13295 tablet_service.cc:1467] Processing CreateTablet for tablet df28831df92d4c12b63b2f7da6c6449a (DEFAULT_TABLE table=rebalance_test_table_1 [id=ba90ff8253204f25829d57a56c0b5e94]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:16.372908 14125 ts_tablet_manager.cc:1398] T 7239f94a7c88433aa6062390a54e7562 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:16.373571 13296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 477c33f203f649019eff2b28524fad90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:16.373692 14125 tablet_bootstrap.cc:492] T 477c33f203f649019eff2b28524fad90 P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:16.374043 13295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df28831df92d4c12b63b2f7da6c6449a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:16.374548 14125 tablet_bootstrap.cc:654] T 477c33f203f649019eff2b28524fad90 P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:16.375177 14130 tablet_bootstrap.cc:492] T df28831df92d4c12b63b2f7da6c6449a P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:16.375454 13164 tablet_service.cc:1467] Processing CreateTablet for tablet df28831df92d4c12b63b2f7da6c6449a (DEFAULT_TABLE table=rebalance_test_table_1 [id=ba90ff8253204f25829d57a56c0b5e94]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:16.375674 13164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df28831df92d4c12b63b2f7da6c6449a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:16.375990 14130 tablet_bootstrap.cc:654] T df28831df92d4c12b63b2f7da6c6449a P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:16.377065 14125 tablet_bootstrap.cc:492] T 477c33f203f649019eff2b28524fad90 P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:16.377147 14125 ts_tablet_manager.cc:1367] T 477c33f203f649019eff2b28524fad90 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 13:56:16.377594 14125 raft_consensus.cc:348] T 477c33f203f649019eff2b28524fad90 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:16.377699 14125 raft_consensus.cc:374] T 477c33f203f649019eff2b28524fad90 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:16.377732 14125 raft_consensus.cc:729] T 477c33f203f649019eff2b28524fad90 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:16.377833 14125 consensus_queue.cc:260] T 477c33f203f649019eff2b28524fad90 P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:16.378116 14125 ts_tablet_manager.cc:1398] T 477c33f203f649019eff2b28524fad90 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:16.378506 14126 ts_tablet_manager.cc:1398] T 7239f94a7c88433aa6062390a54e7562 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240503 13:56:16.378856 14127 tablet_bootstrap.cc:492] T 7239f94a7c88433aa6062390a54e7562 P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:16.379313 14126 tablet_bootstrap.cc:492] T 477c33f203f649019eff2b28524fad90 P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:16.380038 14127 tablet_bootstrap.cc:654] T 7239f94a7c88433aa6062390a54e7562 P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:16.380761 14126 tablet_bootstrap.cc:654] T 477c33f203f649019eff2b28524fad90 P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:16.381291 14130 tablet_bootstrap.cc:492] T df28831df92d4c12b63b2f7da6c6449a P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:16.381407 14130 ts_tablet_manager.cc:1367] T df28831df92d4c12b63b2f7da6c6449a P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20240503 13:56:16.381625 14127 tablet_bootstrap.cc:492] T 7239f94a7c88433aa6062390a54e7562 P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:16.381717 14127 ts_tablet_manager.cc:1367] T 7239f94a7c88433aa6062390a54e7562 P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:56:16.381857 14043 raft_consensus.cc:482] T 7239f94a7c88433aa6062390a54e7562 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:16.381950 14043 raft_consensus.cc:504] T 7239f94a7c88433aa6062390a54e7562 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:16.382160 14127 raft_consensus.cc:348] T 7239f94a7c88433aa6062390a54e7562 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:16.382267 14127 raft_consensus.cc:374] T 7239f94a7c88433aa6062390a54e7562 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:16.382295 14127 raft_consensus.cc:729] T 7239f94a7c88433aa6062390a54e7562 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:16.382421 14127 consensus_queue.cc:260] T 7239f94a7c88433aa6062390a54e7562 P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:16.382668 14127 ts_tablet_manager.cc:1398] T 7239f94a7c88433aa6062390a54e7562 P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:16.382812 14127 tablet_bootstrap.cc:492] T df28831df92d4c12b63b2f7da6c6449a P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:16.383200 14043 leader_election.cc:288] T 7239f94a7c88433aa6062390a54e7562 P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:16.383690 14127 tablet_bootstrap.cc:654] T df28831df92d4c12b63b2f7da6c6449a P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:16.384011 14130 raft_consensus.cc:348] T df28831df92d4c12b63b2f7da6c6449a P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:16.384119 14130 raft_consensus.cc:374] T df28831df92d4c12b63b2f7da6c6449a P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:16.384151 14130 raft_consensus.cc:729] T df28831df92d4c12b63b2f7da6c6449a P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:16.384277 14130 consensus_queue.cc:260] T df28831df92d4c12b63b2f7da6c6449a P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:16.384896 14127 tablet_bootstrap.cc:492] T df28831df92d4c12b63b2f7da6c6449a P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:16.384970 14130 ts_tablet_manager.cc:1398] T df28831df92d4c12b63b2f7da6c6449a P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 13:56:16.384984 14127 ts_tablet_manager.cc:1367] T df28831df92d4c12b63b2f7da6c6449a P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:16.385442 14127 raft_consensus.cc:348] T df28831df92d4c12b63b2f7da6c6449a P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:16.385541 14127 raft_consensus.cc:374] T df28831df92d4c12b63b2f7da6c6449a P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:16.385571 14127 raft_consensus.cc:729] T df28831df92d4c12b63b2f7da6c6449a P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:16.385715 14127 consensus_queue.cc:260] T df28831df92d4c12b63b2f7da6c6449a P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:16.386073 14127 ts_tablet_manager.cc:1398] T df28831df92d4c12b63b2f7da6c6449a P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:16.386233 14127 tablet_bootstrap.cc:492] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:16.387243 14127 tablet_bootstrap.cc:654] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:16.388146 14126 tablet_bootstrap.cc:492] T 477c33f203f649019eff2b28524fad90 P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:16.388242 14126 ts_tablet_manager.cc:1367] T 477c33f203f649019eff2b28524fad90 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240503 13:56:16.388731 14126 raft_consensus.cc:348] T 477c33f203f649019eff2b28524fad90 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:16.388832 14126 raft_consensus.cc:374] T 477c33f203f649019eff2b28524fad90 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:16.388864 14126 raft_consensus.cc:729] T 477c33f203f649019eff2b28524fad90 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:16.389021 14126 consensus_queue.cc:260] T 477c33f203f649019eff2b28524fad90 P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:16.389626 14126 ts_tablet_manager.cc:1398] T 477c33f203f649019eff2b28524fad90 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:16.389793 14126 tablet_bootstrap.cc:492] T df28831df92d4c12b63b2f7da6c6449a P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:16.390642 13315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7239f94a7c88433aa6062390a54e7562" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703" is_pre_election: true
I20240503 13:56:16.390739 14126 tablet_bootstrap.cc:654] T df28831df92d4c12b63b2f7da6c6449a P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:16.390765 13315 raft_consensus.cc:2454] T 7239f94a7c88433aa6062390a54e7562 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 0.
I20240503 13:56:16.391108 13121 leader_election.cc:302] T 7239f94a7c88433aa6062390a54e7562 P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ccf5a4ac450414c9d3579bd5800b770, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:16.391340 14043 raft_consensus.cc:2781] T 7239f94a7c88433aa6062390a54e7562 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:16.391387 14043 raft_consensus.cc:482] T 7239f94a7c88433aa6062390a54e7562 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:16.391422 14043 raft_consensus.cc:3037] T 7239f94a7c88433aa6062390a54e7562 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:16.391873 14127 tablet_bootstrap.cc:492] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:16.391961 14127 ts_tablet_manager.cc:1367] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 13:56:16.392395 14043 raft_consensus.cc:504] T 7239f94a7c88433aa6062390a54e7562 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:16.392395 14127 raft_consensus.cc:348] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:16.392516 14127 raft_consensus.cc:374] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:16.392549 14127 raft_consensus.cc:729] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:16.392673 14127 consensus_queue.cc:260] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:16.392838 14043 leader_election.cc:288] T 7239f94a7c88433aa6062390a54e7562 P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 election: Requested vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:16.392941 14127 ts_tablet_manager.cc:1398] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:16.393199 13315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7239f94a7c88433aa6062390a54e7562" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703"
I20240503 13:56:16.393311 13315 raft_consensus.cc:3037] T 7239f94a7c88433aa6062390a54e7562 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:16.393677 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7239f94a7c88433aa6062390a54e7562" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
I20240503 13:56:16.393877 13447 raft_consensus.cc:2454] T 7239f94a7c88433aa6062390a54e7562 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 0.
I20240503 13:56:16.394134 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7239f94a7c88433aa6062390a54e7562" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
I20240503 13:56:16.394232 13446 raft_consensus.cc:3037] T 7239f94a7c88433aa6062390a54e7562 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:16.394232 14126 tablet_bootstrap.cc:492] T df28831df92d4c12b63b2f7da6c6449a P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:16.394248 13315 raft_consensus.cc:2454] T 7239f94a7c88433aa6062390a54e7562 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 1.
I20240503 13:56:16.394315 14126 ts_tablet_manager.cc:1367] T df28831df92d4c12b63b2f7da6c6449a P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240503 13:56:16.394745 14126 raft_consensus.cc:348] T df28831df92d4c12b63b2f7da6c6449a P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:16.394749 13121 leader_election.cc:302] T 7239f94a7c88433aa6062390a54e7562 P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ccf5a4ac450414c9d3579bd5800b770, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:16.394855 14126 raft_consensus.cc:374] T df28831df92d4c12b63b2f7da6c6449a P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:16.394881 14126 raft_consensus.cc:729] T df28831df92d4c12b63b2f7da6c6449a P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:16.394948 14043 raft_consensus.cc:2781] T 7239f94a7c88433aa6062390a54e7562 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:16.394995 14126 consensus_queue.cc:260] T df28831df92d4c12b63b2f7da6c6449a P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:16.395128 14043 raft_consensus.cc:686] T 7239f94a7c88433aa6062390a54e7562 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 LEADER]: Becoming Leader. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Running, Role: LEADER
I20240503 13:56:16.395179 13446 raft_consensus.cc:2454] T 7239f94a7c88433aa6062390a54e7562 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 1.
I20240503 13:56:16.395313 14126 ts_tablet_manager.cc:1398] T df28831df92d4c12b63b2f7da6c6449a P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:16.395408 14043 consensus_queue.cc:237] T 7239f94a7c88433aa6062390a54e7562 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:16.397356 12906 catalog_manager.cc:5484] T 7239f94a7c88433aa6062390a54e7562 P 8ccf5a4ac450414c9d3579bd5800b770 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66). New cstate: current_term: 1 leader_uuid: "8ccf5a4ac450414c9d3579bd5800b770" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:16.500479 14142 raft_consensus.cc:482] T 477c33f203f649019eff2b28524fad90 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:16.500634 14142 raft_consensus.cc:504] T 477c33f203f649019eff2b28524fad90 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:16.501108 14142 leader_election.cc:288] T 477c33f203f649019eff2b28524fad90 P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:16.501554 13315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "477c33f203f649019eff2b28524fad90" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703" is_pre_election: true
I20240503 13:56:16.501559 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "477c33f203f649019eff2b28524fad90" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
I20240503 13:56:16.501680 13315 raft_consensus.cc:2454] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 0.
I20240503 13:56:16.501714 13446 raft_consensus.cc:2454] T 477c33f203f649019eff2b28524fad90 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 0.
I20240503 13:56:16.501965 13121 leader_election.cc:302] T 477c33f203f649019eff2b28524fad90 P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ccf5a4ac450414c9d3579bd5800b770, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:16.502148 14142 raft_consensus.cc:2781] T 477c33f203f649019eff2b28524fad90 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:16.502202 14142 raft_consensus.cc:482] T 477c33f203f649019eff2b28524fad90 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:16.502228 14142 raft_consensus.cc:3037] T 477c33f203f649019eff2b28524fad90 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:16.503036 14142 raft_consensus.cc:504] T 477c33f203f649019eff2b28524fad90 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:16.503346 14142 leader_election.cc:288] T 477c33f203f649019eff2b28524fad90 P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 election: Requested vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:16.503695 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "477c33f203f649019eff2b28524fad90" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
I20240503 13:56:16.503695 13315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "477c33f203f649019eff2b28524fad90" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703"
I20240503 13:56:16.503819 13315 raft_consensus.cc:3037] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:16.503819 13446 raft_consensus.cc:3037] T 477c33f203f649019eff2b28524fad90 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:16.504819 13315 raft_consensus.cc:2454] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 1.
I20240503 13:56:16.504819 13446 raft_consensus.cc:2454] T 477c33f203f649019eff2b28524fad90 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 1.
I20240503 13:56:16.505165 13121 leader_election.cc:302] T 477c33f203f649019eff2b28524fad90 P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ccf5a4ac450414c9d3579bd5800b770, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:16.505342 14142 raft_consensus.cc:2781] T 477c33f203f649019eff2b28524fad90 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:16.505425 14142 raft_consensus.cc:686] T 477c33f203f649019eff2b28524fad90 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 LEADER]: Becoming Leader. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Running, Role: LEADER
I20240503 13:56:16.505564 14142 consensus_queue.cc:237] T 477c33f203f649019eff2b28524fad90 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:16.507215 12906 catalog_manager.cc:5484] T 477c33f203f649019eff2b28524fad90 P 8ccf5a4ac450414c9d3579bd5800b770 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66). New cstate: current_term: 1 leader_uuid: "8ccf5a4ac450414c9d3579bd5800b770" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:16.587373 14142 raft_consensus.cc:482] T df28831df92d4c12b63b2f7da6c6449a P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:16.587519 14142 raft_consensus.cc:504] T df28831df92d4c12b63b2f7da6c6449a P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:16.588100 14142 leader_election.cc:288] T df28831df92d4c12b63b2f7da6c6449a P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695), ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:16.588682 13315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df28831df92d4c12b63b2f7da6c6449a" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703" is_pre_election: true
I20240503 13:56:16.588840 13315 raft_consensus.cc:2454] T df28831df92d4c12b63b2f7da6c6449a P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 0.
I20240503 13:56:16.589133 13121 leader_election.cc:302] T df28831df92d4c12b63b2f7da6c6449a P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ccf5a4ac450414c9d3579bd5800b770, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:16.589315 14142 raft_consensus.cc:2781] T df28831df92d4c12b63b2f7da6c6449a P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:16.589378 14142 raft_consensus.cc:482] T df28831df92d4c12b63b2f7da6c6449a P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:16.589416 14142 raft_consensus.cc:3037] T df28831df92d4c12b63b2f7da6c6449a P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:16.590344 14142 raft_consensus.cc:504] T df28831df92d4c12b63b2f7da6c6449a P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:16.590719 14142 leader_election.cc:288] T df28831df92d4c12b63b2f7da6c6449a P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 election: Requested vote from peers 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695), ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:16.591207 13315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df28831df92d4c12b63b2f7da6c6449a" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703"
I20240503 13:56:16.591346 13315 raft_consensus.cc:3037] T df28831df92d4c12b63b2f7da6c6449a P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:16.591761 13054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df28831df92d4c12b63b2f7da6c6449a" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e" is_pre_election: true
I20240503 13:56:16.591830 13053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df28831df92d4c12b63b2f7da6c6449a" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e"
I20240503 13:56:16.591892 13054 raft_consensus.cc:2454] T df28831df92d4c12b63b2f7da6c6449a P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 0.
I20240503 13:56:16.591958 13053 raft_consensus.cc:3037] T df28831df92d4c12b63b2f7da6c6449a P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:16.592280 13315 raft_consensus.cc:2454] T df28831df92d4c12b63b2f7da6c6449a P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 1.
I20240503 13:56:16.592579 13121 leader_election.cc:302] T df28831df92d4c12b63b2f7da6c6449a P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ccf5a4ac450414c9d3579bd5800b770, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:16.592743 14142 raft_consensus.cc:2781] T df28831df92d4c12b63b2f7da6c6449a P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:16.592832 14142 raft_consensus.cc:686] T df28831df92d4c12b63b2f7da6c6449a P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 LEADER]: Becoming Leader. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Running, Role: LEADER
I20240503 13:56:16.592912 13053 raft_consensus.cc:2454] T df28831df92d4c12b63b2f7da6c6449a P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 1.
I20240503 13:56:16.592972 14142 consensus_queue.cc:237] T df28831df92d4c12b63b2f7da6c6449a P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:16.594556 12906 catalog_manager.cc:5484] T df28831df92d4c12b63b2f7da6c6449a P 8ccf5a4ac450414c9d3579bd5800b770 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66). New cstate: current_term: 1 leader_uuid: "8ccf5a4ac450414c9d3579bd5800b770" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } }
W20240503 13:56:16.714783 14146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:56:16.715034 14146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:56:16.718686 14146 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240503 13:56:16.802151 13315 raft_consensus.cc:1261] T 7239f94a7c88433aa6062390a54e7562 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Refusing update from remote peer 8ccf5a4ac450414c9d3579bd5800b770: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:16.802877 13446 raft_consensus.cc:1261] T 7239f94a7c88433aa6062390a54e7562 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer 8ccf5a4ac450414c9d3579bd5800b770: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:16.802878 13447 raft_consensus.cc:1261] T 477c33f203f649019eff2b28524fad90 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer 8ccf5a4ac450414c9d3579bd5800b770: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:16.803825 13053 raft_consensus.cc:1261] T df28831df92d4c12b63b2f7da6c6449a P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Refusing update from remote peer 8ccf5a4ac450414c9d3579bd5800b770: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:16.804476 13316 raft_consensus.cc:1261] T df28831df92d4c12b63b2f7da6c6449a P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Refusing update from remote peer 8ccf5a4ac450414c9d3579bd5800b770: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:16.804750 14141 consensus_queue.cc:1035] T 7239f94a7c88433aa6062390a54e7562 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:16.805720 14142 consensus_queue.cc:1035] T df28831df92d4c12b63b2f7da6c6449a P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:16.805912 13315 raft_consensus.cc:1261] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Refusing update from remote peer 8ccf5a4ac450414c9d3579bd5800b770: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:16.806013 14142 consensus_queue.cc:1035] T 7239f94a7c88433aa6062390a54e7562 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:16.806167 14043 consensus_queue.cc:1035] T 477c33f203f649019eff2b28524fad90 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:16.806866 14043 consensus_queue.cc:1035] T df28831df92d4c12b63b2f7da6c6449a P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:16.807159 14043 consensus_queue.cc:1035] T 477c33f203f649019eff2b28524fad90 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 93.1082 ms
time per row: 0.0465541 ms
I20240503 13:56:16.864250 12906 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:56:16.870332 13033 tablet_service.cc:1467] Processing CreateTablet for tablet 1be7152bed00461aa66db3ef430d7460 (DEFAULT_TABLE table=rebalance_test_table_2 [id=16147ff5bb1c4ed9a5ab17905cb020ca]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:16.870412 13164 tablet_service.cc:1467] Processing CreateTablet for tablet 1be7152bed00461aa66db3ef430d7460 (DEFAULT_TABLE table=rebalance_test_table_2 [id=16147ff5bb1c4ed9a5ab17905cb020ca]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:16.870676 13033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1be7152bed00461aa66db3ef430d7460. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:16.870707 13164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1be7152bed00461aa66db3ef430d7460. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:16.871060 13034 tablet_service.cc:1467] Processing CreateTablet for tablet b9f328a343734ca89808aef24928c41b (DEFAULT_TABLE table=rebalance_test_table_2 [id=16147ff5bb1c4ed9a5ab17905cb020ca]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:16.871271 13034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9f328a343734ca89808aef24928c41b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:16.872540 14126 tablet_bootstrap.cc:492] T 1be7152bed00461aa66db3ef430d7460 P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:16.872749 13032 tablet_service.cc:1467] Processing CreateTablet for tablet b35569e6e1b34328bf334fe84551193b (DEFAULT_TABLE table=rebalance_test_table_2 [id=16147ff5bb1c4ed9a5ab17905cb020ca]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:16.872818 14130 tablet_bootstrap.cc:492] T 1be7152bed00461aa66db3ef430d7460 P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:16.872980 13032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b35569e6e1b34328bf334fe84551193b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:16.873397 14126 tablet_bootstrap.cc:654] T 1be7152bed00461aa66db3ef430d7460 P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:16.873658 14130 tablet_bootstrap.cc:654] T 1be7152bed00461aa66db3ef430d7460 P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:16.874538 14126 tablet_bootstrap.cc:492] T 1be7152bed00461aa66db3ef430d7460 P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:16.874625 14126 ts_tablet_manager.cc:1367] T 1be7152bed00461aa66db3ef430d7460 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:56:16.874660 13296 tablet_service.cc:1467] Processing CreateTablet for tablet b35569e6e1b34328bf334fe84551193b (DEFAULT_TABLE table=rebalance_test_table_2 [id=16147ff5bb1c4ed9a5ab17905cb020ca]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:16.874923 13296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b35569e6e1b34328bf334fe84551193b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:16.875082 14126 raft_consensus.cc:348] T 1be7152bed00461aa66db3ef430d7460 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:16.875190 14126 raft_consensus.cc:374] T 1be7152bed00461aa66db3ef430d7460 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:16.875221 14126 raft_consensus.cc:729] T 1be7152bed00461aa66db3ef430d7460 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:16.875347 14126 consensus_queue.cc:260] T 1be7152bed00461aa66db3ef430d7460 P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:16.875352 13165 tablet_service.cc:1467] Processing CreateTablet for tablet b9f328a343734ca89808aef24928c41b (DEFAULT_TABLE table=rebalance_test_table_2 [id=16147ff5bb1c4ed9a5ab17905cb020ca]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:16.875636 14126 ts_tablet_manager.cc:1398] T 1be7152bed00461aa66db3ef430d7460 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:16.875649 13165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9f328a343734ca89808aef24928c41b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:16.875945 13426 tablet_service.cc:1467] Processing CreateTablet for tablet 1be7152bed00461aa66db3ef430d7460 (DEFAULT_TABLE table=rebalance_test_table_2 [id=16147ff5bb1c4ed9a5ab17905cb020ca]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:16.876207 13426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1be7152bed00461aa66db3ef430d7460. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:16.876703 13427 tablet_service.cc:1467] Processing CreateTablet for tablet b35569e6e1b34328bf334fe84551193b (DEFAULT_TABLE table=rebalance_test_table_2 [id=16147ff5bb1c4ed9a5ab17905cb020ca]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:16.876914 13427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b35569e6e1b34328bf334fe84551193b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:16.877537 14126 tablet_bootstrap.cc:492] T b9f328a343734ca89808aef24928c41b P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:16.877952 14127 tablet_bootstrap.cc:492] T b35569e6e1b34328bf334fe84551193b P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:16.878414 14126 tablet_bootstrap.cc:654] T b9f328a343734ca89808aef24928c41b P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:16.878737 14127 tablet_bootstrap.cc:654] T b35569e6e1b34328bf334fe84551193b P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:16.878957 13425 tablet_service.cc:1467] Processing CreateTablet for tablet b9f328a343734ca89808aef24928c41b (DEFAULT_TABLE table=rebalance_test_table_2 [id=16147ff5bb1c4ed9a5ab17905cb020ca]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:16.879148 13425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9f328a343734ca89808aef24928c41b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:16.879544 14126 tablet_bootstrap.cc:492] T b9f328a343734ca89808aef24928c41b P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:16.879627 14126 ts_tablet_manager.cc:1367] T b9f328a343734ca89808aef24928c41b P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:16.880007 14130 tablet_bootstrap.cc:492] T 1be7152bed00461aa66db3ef430d7460 P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:16.880105 14130 ts_tablet_manager.cc:1367] T 1be7152bed00461aa66db3ef430d7460 P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20240503 13:56:16.880056 14126 raft_consensus.cc:348] T b9f328a343734ca89808aef24928c41b P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:16.880156 14126 raft_consensus.cc:374] T b9f328a343734ca89808aef24928c41b P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:16.880185 14126 raft_consensus.cc:729] T b9f328a343734ca89808aef24928c41b P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:16.880338 14126 consensus_queue.cc:260] T b9f328a343734ca89808aef24928c41b P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:16.880578 14130 raft_consensus.cc:348] T 1be7152bed00461aa66db3ef430d7460 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:16.880689 14130 raft_consensus.cc:374] T 1be7152bed00461aa66db3ef430d7460 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:16.880698 14126 ts_tablet_manager.cc:1398] T b9f328a343734ca89808aef24928c41b P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:16.880719 14130 raft_consensus.cc:729] T 1be7152bed00461aa66db3ef430d7460 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:16.880837 14130 consensus_queue.cc:260] T 1be7152bed00461aa66db3ef430d7460 P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:16.881114 14130 ts_tablet_manager.cc:1398] T 1be7152bed00461aa66db3ef430d7460 P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:16.881899 14127 tablet_bootstrap.cc:492] T b35569e6e1b34328bf334fe84551193b P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:16.881992 14127 ts_tablet_manager.cc:1367] T b35569e6e1b34328bf334fe84551193b P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 13:56:16.882429 14127 raft_consensus.cc:348] T b35569e6e1b34328bf334fe84551193b P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:16.882545 14127 raft_consensus.cc:374] T b35569e6e1b34328bf334fe84551193b P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:16.882598 14127 raft_consensus.cc:729] T b35569e6e1b34328bf334fe84551193b P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:16.882726 14127 consensus_queue.cc:260] T b35569e6e1b34328bf334fe84551193b P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:16.882998 14127 ts_tablet_manager.cc:1398] T b35569e6e1b34328bf334fe84551193b P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:16.883414 14185 tablet_bootstrap.cc:492] T b35569e6e1b34328bf334fe84551193b P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:16.884231 14185 tablet_bootstrap.cc:654] T b35569e6e1b34328bf334fe84551193b P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:16.881388 14130 tablet_bootstrap.cc:492] T b9f328a343734ca89808aef24928c41b P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:16.885370 14130 tablet_bootstrap.cc:654] T b9f328a343734ca89808aef24928c41b P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:16.885744 14185 tablet_bootstrap.cc:492] T b35569e6e1b34328bf334fe84551193b P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:16.885826 14185 ts_tablet_manager.cc:1367] T b35569e6e1b34328bf334fe84551193b P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:56:16.886235 14185 raft_consensus.cc:348] T b35569e6e1b34328bf334fe84551193b P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:16.886348 14185 raft_consensus.cc:374] T b35569e6e1b34328bf334fe84551193b P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:16.886377 14185 raft_consensus.cc:729] T b35569e6e1b34328bf334fe84551193b P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:16.886395 14130 tablet_bootstrap.cc:492] T b9f328a343734ca89808aef24928c41b P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:16.886468 14130 ts_tablet_manager.cc:1367] T b9f328a343734ca89808aef24928c41b P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 13:56:16.886519 14185 consensus_queue.cc:260] T b35569e6e1b34328bf334fe84551193b P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:16.886871 14130 raft_consensus.cc:348] T b9f328a343734ca89808aef24928c41b P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:16.886967 14130 raft_consensus.cc:374] T b9f328a343734ca89808aef24928c41b P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:16.886988 14130 raft_consensus.cc:729] T b9f328a343734ca89808aef24928c41b P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:16.887140 14185 ts_tablet_manager.cc:1398] T b35569e6e1b34328bf334fe84551193b P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:16.887140 14130 consensus_queue.cc:260] T b9f328a343734ca89808aef24928c41b P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:16.887308 14185 tablet_bootstrap.cc:492] T b9f328a343734ca89808aef24928c41b P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:16.888065 14185 tablet_bootstrap.cc:654] T b9f328a343734ca89808aef24928c41b P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:16.888701 14130 ts_tablet_manager.cc:1398] T b9f328a343734ca89808aef24928c41b P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:16.888994 14185 tablet_bootstrap.cc:492] T b9f328a343734ca89808aef24928c41b P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:16.889066 14185 ts_tablet_manager.cc:1367] T b9f328a343734ca89808aef24928c41b P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:56:16.889303 14130 tablet_bootstrap.cc:492] T b35569e6e1b34328bf334fe84551193b P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:16.889624 14185 raft_consensus.cc:348] T b9f328a343734ca89808aef24928c41b P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:16.889736 14185 raft_consensus.cc:374] T b9f328a343734ca89808aef24928c41b P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:16.889775 14185 raft_consensus.cc:729] T b9f328a343734ca89808aef24928c41b P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:16.889875 14185 consensus_queue.cc:260] T b9f328a343734ca89808aef24928c41b P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:16.890110 14185 ts_tablet_manager.cc:1398] T b9f328a343734ca89808aef24928c41b P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:16.890204 14130 tablet_bootstrap.cc:654] T b35569e6e1b34328bf334fe84551193b P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:16.890311 14185 tablet_bootstrap.cc:492] T 1be7152bed00461aa66db3ef430d7460 P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:16.891116 14185 tablet_bootstrap.cc:654] T 1be7152bed00461aa66db3ef430d7460 P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:16.891399 14130 tablet_bootstrap.cc:492] T b35569e6e1b34328bf334fe84551193b P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:16.891470 14130 ts_tablet_manager.cc:1367] T b35569e6e1b34328bf334fe84551193b P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:16.891827 14130 raft_consensus.cc:348] T b35569e6e1b34328bf334fe84551193b P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:16.891927 14130 raft_consensus.cc:374] T b35569e6e1b34328bf334fe84551193b P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:16.891960 14130 raft_consensus.cc:729] T b35569e6e1b34328bf334fe84551193b P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:16.892069 14130 consensus_queue.cc:260] T b35569e6e1b34328bf334fe84551193b P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:16.892604 14130 ts_tablet_manager.cc:1398] T b35569e6e1b34328bf334fe84551193b P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:16.892964 14185 tablet_bootstrap.cc:492] T 1be7152bed00461aa66db3ef430d7460 P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:16.893045 14185 ts_tablet_manager.cc:1367] T 1be7152bed00461aa66db3ef430d7460 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 13:56:16.893563 14185 raft_consensus.cc:348] T 1be7152bed00461aa66db3ef430d7460 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:16.893646 14185 raft_consensus.cc:374] T 1be7152bed00461aa66db3ef430d7460 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:16.893671 14185 raft_consensus.cc:729] T 1be7152bed00461aa66db3ef430d7460 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:16.893783 14185 consensus_queue.cc:260] T 1be7152bed00461aa66db3ef430d7460 P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:16.894032 14185 ts_tablet_manager.cc:1398] T 1be7152bed00461aa66db3ef430d7460 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:16.898977 14043 raft_consensus.cc:482] T 1be7152bed00461aa66db3ef430d7460 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:16.899098 14043 raft_consensus.cc:504] T 1be7152bed00461aa66db3ef430d7460 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:16.899477 14043 leader_election.cc:288] T 1be7152bed00461aa66db3ef430d7460 P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695), 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:16.899814 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1be7152bed00461aa66db3ef430d7460" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
I20240503 13:56:16.899832 13053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1be7152bed00461aa66db3ef430d7460" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e" is_pre_election: true
I20240503 13:56:16.899935 13447 raft_consensus.cc:2454] T 1be7152bed00461aa66db3ef430d7460 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 0.
I20240503 13:56:16.899951 13053 raft_consensus.cc:2454] T 1be7152bed00461aa66db3ef430d7460 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 0.
I20240503 13:56:16.900240 13118 leader_election.cc:302] T 1be7152bed00461aa66db3ef430d7460 P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, 8ccf5a4ac450414c9d3579bd5800b770; no voters:
I20240503 13:56:16.900427 14043 raft_consensus.cc:2781] T 1be7152bed00461aa66db3ef430d7460 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:16.900521 14043 raft_consensus.cc:482] T 1be7152bed00461aa66db3ef430d7460 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:16.900568 14043 raft_consensus.cc:3037] T 1be7152bed00461aa66db3ef430d7460 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:16.901422 14043 raft_consensus.cc:504] T 1be7152bed00461aa66db3ef430d7460 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:16.901773 14043 leader_election.cc:288] T 1be7152bed00461aa66db3ef430d7460 P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 election: Requested vote from peers 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695), 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:16.902081 13054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1be7152bed00461aa66db3ef430d7460" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e"
I20240503 13:56:16.902137 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1be7152bed00461aa66db3ef430d7460" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
I20240503 13:56:16.902225 13054 raft_consensus.cc:3037] T 1be7152bed00461aa66db3ef430d7460 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:16.902240 13447 raft_consensus.cc:3037] T 1be7152bed00461aa66db3ef430d7460 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:16.902927 13447 raft_consensus.cc:2454] T 1be7152bed00461aa66db3ef430d7460 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 1.
I20240503 13:56:16.902936 13054 raft_consensus.cc:2454] T 1be7152bed00461aa66db3ef430d7460 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 1.
I20240503 13:56:16.903192 13118 leader_election.cc:302] T 1be7152bed00461aa66db3ef430d7460 P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, 8ccf5a4ac450414c9d3579bd5800b770; no voters:
I20240503 13:56:16.903421 14043 raft_consensus.cc:2781] T 1be7152bed00461aa66db3ef430d7460 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:16.903507 14043 raft_consensus.cc:686] T 1be7152bed00461aa66db3ef430d7460 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 LEADER]: Becoming Leader. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Running, Role: LEADER
I20240503 13:56:16.903640 14043 consensus_queue.cc:237] T 1be7152bed00461aa66db3ef430d7460 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:16.905220 12911 catalog_manager.cc:5484] T 1be7152bed00461aa66db3ef430d7460 P 8ccf5a4ac450414c9d3579bd5800b770 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66). New cstate: current_term: 1 leader_uuid: "8ccf5a4ac450414c9d3579bd5800b770" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } }
I20240503 13:56:16.932796 14100 raft_consensus.cc:482] T b9f328a343734ca89808aef24928c41b P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:16.932934 14100 raft_consensus.cc:504] T b9f328a343734ca89808aef24928c41b P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:16.933360 14100 leader_election.cc:288] T b9f328a343734ca89808aef24928c41b P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:16.933833 13054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9f328a343734ca89808aef24928c41b" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e" is_pre_election: true
I20240503 13:56:16.933915 13184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9f328a343734ca89808aef24928c41b" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770" is_pre_election: true
I20240503 13:56:16.933984 13054 raft_consensus.cc:2454] T b9f328a343734ca89808aef24928c41b P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 0.
I20240503 13:56:16.934062 13184 raft_consensus.cc:2454] T b9f328a343734ca89808aef24928c41b P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 0.
I20240503 13:56:16.934273 13381 leader_election.cc:302] T b9f328a343734ca89808aef24928c41b P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 7690ee79577c45e3a2c8f8b552c0bff8; no voters:
I20240503 13:56:16.934433 14100 raft_consensus.cc:2781] T b9f328a343734ca89808aef24928c41b P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:16.934492 14100 raft_consensus.cc:482] T b9f328a343734ca89808aef24928c41b P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:16.934537 14100 raft_consensus.cc:3037] T b9f328a343734ca89808aef24928c41b P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:16.935644 14100 raft_consensus.cc:504] T b9f328a343734ca89808aef24928c41b P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:16.935983 14100 leader_election.cc:288] T b9f328a343734ca89808aef24928c41b P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 election: Requested vote from peers 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:16.936308 13054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9f328a343734ca89808aef24928c41b" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e"
I20240503 13:56:16.936417 13054 raft_consensus.cc:3037] T b9f328a343734ca89808aef24928c41b P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:16.936385 13184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9f328a343734ca89808aef24928c41b" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770"
I20240503 13:56:16.936566 13184 raft_consensus.cc:3037] T b9f328a343734ca89808aef24928c41b P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:16.937327 13054 raft_consensus.cc:2454] T b9f328a343734ca89808aef24928c41b P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 1.
I20240503 13:56:16.937569 13184 raft_consensus.cc:2454] T b9f328a343734ca89808aef24928c41b P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 1.
I20240503 13:56:16.937908 13383 leader_election.cc:302] T b9f328a343734ca89808aef24928c41b P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, 8ccf5a4ac450414c9d3579bd5800b770; no voters:
I20240503 13:56:16.938413 14100 raft_consensus.cc:2781] T b9f328a343734ca89808aef24928c41b P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:16.938501 14100 raft_consensus.cc:686] T b9f328a343734ca89808aef24928c41b P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 LEADER]: Becoming Leader. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Running, Role: LEADER
I20240503 13:56:16.938638 14100 consensus_queue.cc:237] T b9f328a343734ca89808aef24928c41b P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:16.938830 14113 raft_consensus.cc:482] T b35569e6e1b34328bf334fe84551193b P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:16.938982 14113 raft_consensus.cc:504] T b35569e6e1b34328bf334fe84551193b P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:16.940235 14113 leader_election.cc:288] T b35569e6e1b34328bf334fe84551193b P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695), 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:16.940779 12911 catalog_manager.cc:5484] T b9f328a343734ca89808aef24928c41b P 7690ee79577c45e3a2c8f8b552c0bff8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68). New cstate: current_term: 1 leader_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } }
I20240503 13:56:16.945539 13054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b35569e6e1b34328bf334fe84551193b" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e" is_pre_election: true
I20240503 13:56:16.945727 13054 raft_consensus.cc:2454] T b35569e6e1b34328bf334fe84551193b P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 0.
I20240503 13:56:16.945765 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b35569e6e1b34328bf334fe84551193b" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
I20240503 13:56:16.945905 13447 raft_consensus.cc:2454] T b35569e6e1b34328bf334fe84551193b P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 0.
I20240503 13:56:16.946173 13250 leader_election.cc:302] T b35569e6e1b34328bf334fe84551193b P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:16.946502 14113 raft_consensus.cc:2781] T b35569e6e1b34328bf334fe84551193b P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:16.946575 14113 raft_consensus.cc:482] T b35569e6e1b34328bf334fe84551193b P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:16.946624 14113 raft_consensus.cc:3037] T b35569e6e1b34328bf334fe84551193b P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:16.947458 14113 raft_consensus.cc:504] T b35569e6e1b34328bf334fe84551193b P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:16.947887 14113 leader_election.cc:288] T b35569e6e1b34328bf334fe84551193b P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: Requested vote from peers 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695), 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:16.948277 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b35569e6e1b34328bf334fe84551193b" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
I20240503 13:56:16.948266 13054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b35569e6e1b34328bf334fe84551193b" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e"
I20240503 13:56:16.948521 13447 raft_consensus.cc:3037] T b35569e6e1b34328bf334fe84551193b P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:16.948522 13054 raft_consensus.cc:3037] T b35569e6e1b34328bf334fe84551193b P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:16.949285 13054 raft_consensus.cc:2454] T b35569e6e1b34328bf334fe84551193b P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 1.
I20240503 13:56:16.949326 13447 raft_consensus.cc:2454] T b35569e6e1b34328bf334fe84551193b P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 1.
I20240503 13:56:16.949569 13250 leader_election.cc:302] T b35569e6e1b34328bf334fe84551193b P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:16.949756 14113 raft_consensus.cc:2781] T b35569e6e1b34328bf334fe84551193b P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:16.949847 14113 raft_consensus.cc:686] T b35569e6e1b34328bf334fe84551193b P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Becoming Leader. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Running, Role: LEADER
I20240503 13:56:16.950156 14113 consensus_queue.cc:237] T b35569e6e1b34328bf334fe84551193b P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:16.952224 12911 catalog_manager.cc:5484] T b35569e6e1b34328bf334fe84551193b P ba0194098aa84449ae17e8a0d4c43703 reported cstate change: term changed from 0 to 1, leader changed from <none> to ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67). New cstate: current_term: 1 leader_uuid: "ba0194098aa84449ae17e8a0d4c43703" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
W20240503 13:56:17.071424 14199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:56:17.071695 14199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:56:17.074870 14199 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240503 13:56:17.152909 13184 raft_consensus.cc:1261] T b9f328a343734ca89808aef24928c41b P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Refusing update from remote peer 7690ee79577c45e3a2c8f8b552c0bff8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:17.154105 13447 raft_consensus.cc:1261] T 1be7152bed00461aa66db3ef430d7460 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer 8ccf5a4ac450414c9d3579bd5800b770: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:17.154502 14043 consensus_queue.cc:1035] T 1be7152bed00461aa66db3ef430d7460 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:17.156795 13054 raft_consensus.cc:1261] T b9f328a343734ca89808aef24928c41b P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Refusing update from remote peer 7690ee79577c45e3a2c8f8b552c0bff8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:17.157177 13053 raft_consensus.cc:1261] T 1be7152bed00461aa66db3ef430d7460 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Refusing update from remote peer 8ccf5a4ac450414c9d3579bd5800b770: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:17.157544 14100 consensus_queue.cc:1035] T b9f328a343734ca89808aef24928c41b P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:17.157537 14141 consensus_queue.cc:1035] T 1be7152bed00461aa66db3ef430d7460 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:17.162384 13446 raft_consensus.cc:1261] T b35569e6e1b34328bf334fe84551193b P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:17.162994 14100 consensus_queue.cc:1035] T b9f328a343734ca89808aef24928c41b P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:17.163352 14053 consensus_queue.cc:1035] T b35569e6e1b34328bf334fe84551193b P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:17.164134 13052 raft_consensus.cc:1261] T b35569e6e1b34328bf334fe84551193b P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:56:17.168138 14113 consensus_queue.cc:1035] T b35569e6e1b34328bf334fe84551193b P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 84.5749 ms
time per row: 0.0422875 ms
I20240503 13:56:17.210664 12912 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:56:17.216117 13296 tablet_service.cc:1467] Processing CreateTablet for tablet a49b6f596d724b35b757ee69fe15e5a5 (DEFAULT_TABLE table=rebalance_test_table_3 [id=b600248fbc0b4c71b7c7cd8dd2844bbf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:17.216172 13032 tablet_service.cc:1467] Processing CreateTablet for tablet a49b6f596d724b35b757ee69fe15e5a5 (DEFAULT_TABLE table=rebalance_test_table_3 [id=b600248fbc0b4c71b7c7cd8dd2844bbf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:17.216434 13296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a49b6f596d724b35b757ee69fe15e5a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:17.216516 13032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a49b6f596d724b35b757ee69fe15e5a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:17.216727 13295 tablet_service.cc:1467] Processing CreateTablet for tablet c047c61f42f04484842125847ca677aa (DEFAULT_TABLE table=rebalance_test_table_3 [id=b600248fbc0b4c71b7c7cd8dd2844bbf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:17.216953 13295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c047c61f42f04484842125847ca677aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:17.218091 14127 tablet_bootstrap.cc:492] T a49b6f596d724b35b757ee69fe15e5a5 P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:17.218612 13294 tablet_service.cc:1467] Processing CreateTablet for tablet 4647faf6a457415d8ae2b6691f452eab (DEFAULT_TABLE table=rebalance_test_table_3 [id=b600248fbc0b4c71b7c7cd8dd2844bbf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:17.218796 14130 tablet_bootstrap.cc:492] T a49b6f596d724b35b757ee69fe15e5a5 P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:17.218817 13294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4647faf6a457415d8ae2b6691f452eab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:17.218967 14127 tablet_bootstrap.cc:654] T a49b6f596d724b35b757ee69fe15e5a5 P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:17.219398 14130 tablet_bootstrap.cc:654] T a49b6f596d724b35b757ee69fe15e5a5 P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:17.220089 14127 tablet_bootstrap.cc:492] T a49b6f596d724b35b757ee69fe15e5a5 P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:17.220088 13165 tablet_service.cc:1467] Processing CreateTablet for tablet 4647faf6a457415d8ae2b6691f452eab (DEFAULT_TABLE table=rebalance_test_table_3 [id=b600248fbc0b4c71b7c7cd8dd2844bbf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:17.220160 14127 ts_tablet_manager.cc:1367] T a49b6f596d724b35b757ee69fe15e5a5 P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:56:17.220321 13165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4647faf6a457415d8ae2b6691f452eab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:17.220600 14127 raft_consensus.cc:348] T a49b6f596d724b35b757ee69fe15e5a5 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:17.220698 14127 raft_consensus.cc:374] T a49b6f596d724b35b757ee69fe15e5a5 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:17.220731 14127 raft_consensus.cc:729] T a49b6f596d724b35b757ee69fe15e5a5 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:17.220846 14127 consensus_queue.cc:260] T a49b6f596d724b35b757ee69fe15e5a5 P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:17.221091 14127 ts_tablet_manager.cc:1398] T a49b6f596d724b35b757ee69fe15e5a5 P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:17.221093 14130 tablet_bootstrap.cc:492] T a49b6f596d724b35b757ee69fe15e5a5 P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:17.221168 14130 ts_tablet_manager.cc:1367] T a49b6f596d724b35b757ee69fe15e5a5 P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:17.221194 14127 tablet_bootstrap.cc:492] T c047c61f42f04484842125847ca677aa P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:17.221596 14130 raft_consensus.cc:348] T a49b6f596d724b35b757ee69fe15e5a5 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:17.221715 14130 raft_consensus.cc:374] T a49b6f596d724b35b757ee69fe15e5a5 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:17.221743 14130 raft_consensus.cc:729] T a49b6f596d724b35b757ee69fe15e5a5 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:17.221845 14127 tablet_bootstrap.cc:654] T c047c61f42f04484842125847ca677aa P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:17.221869 14130 consensus_queue.cc:260] T a49b6f596d724b35b757ee69fe15e5a5 P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:17.222151 14130 ts_tablet_manager.cc:1398] T a49b6f596d724b35b757ee69fe15e5a5 P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:17.222435 13164 tablet_service.cc:1467] Processing CreateTablet for tablet a49b6f596d724b35b757ee69fe15e5a5 (DEFAULT_TABLE table=rebalance_test_table_3 [id=b600248fbc0b4c71b7c7cd8dd2844bbf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:17.222643 13164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a49b6f596d724b35b757ee69fe15e5a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:17.222987 14127 tablet_bootstrap.cc:492] T c047c61f42f04484842125847ca677aa P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:17.223054 14127 ts_tablet_manager.cc:1367] T c047c61f42f04484842125847ca677aa P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:56:17.223435 14127 raft_consensus.cc:348] T c047c61f42f04484842125847ca677aa P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:17.223526 14127 raft_consensus.cc:374] T c047c61f42f04484842125847ca677aa P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:17.223556 14127 raft_consensus.cc:729] T c047c61f42f04484842125847ca677aa P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:17.223660 14127 consensus_queue.cc:260] T c047c61f42f04484842125847ca677aa P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:17.223891 14127 ts_tablet_manager.cc:1398] T c047c61f42f04484842125847ca677aa P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:17.224018 14127 tablet_bootstrap.cc:492] T 4647faf6a457415d8ae2b6691f452eab P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:17.224459 14126 tablet_bootstrap.cc:492] T 4647faf6a457415d8ae2b6691f452eab P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:17.225303 14126 tablet_bootstrap.cc:654] T 4647faf6a457415d8ae2b6691f452eab P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:17.225358 14127 tablet_bootstrap.cc:654] T 4647faf6a457415d8ae2b6691f452eab P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:17.225991 13425 tablet_service.cc:1467] Processing CreateTablet for tablet 4647faf6a457415d8ae2b6691f452eab (DEFAULT_TABLE table=rebalance_test_table_3 [id=b600248fbc0b4c71b7c7cd8dd2844bbf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:17.225991 13426 tablet_service.cc:1467] Processing CreateTablet for tablet c047c61f42f04484842125847ca677aa (DEFAULT_TABLE table=rebalance_test_table_3 [id=b600248fbc0b4c71b7c7cd8dd2844bbf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:17.226298 13425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4647faf6a457415d8ae2b6691f452eab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:17.226429 13426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c047c61f42f04484842125847ca677aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:17.226438 14126 tablet_bootstrap.cc:492] T 4647faf6a457415d8ae2b6691f452eab P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:17.226521 14126 ts_tablet_manager.cc:1367] T 4647faf6a457415d8ae2b6691f452eab P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:17.226938 14126 raft_consensus.cc:348] T 4647faf6a457415d8ae2b6691f452eab P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:17.227030 14126 raft_consensus.cc:374] T 4647faf6a457415d8ae2b6691f452eab P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:17.227059 14126 raft_consensus.cc:729] T 4647faf6a457415d8ae2b6691f452eab P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:17.227195 14126 consensus_queue.cc:260] T 4647faf6a457415d8ae2b6691f452eab P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:17.227445 14126 ts_tablet_manager.cc:1398] T 4647faf6a457415d8ae2b6691f452eab P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:17.227602 14126 tablet_bootstrap.cc:492] T a49b6f596d724b35b757ee69fe15e5a5 P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:17.227821 13032 tablet_service.cc:1467] Processing CreateTablet for tablet c047c61f42f04484842125847ca677aa (DEFAULT_TABLE table=rebalance_test_table_3 [id=b600248fbc0b4c71b7c7cd8dd2844bbf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:17.228081 13032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c047c61f42f04484842125847ca677aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:17.228297 14185 tablet_bootstrap.cc:492] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:17.228401 14126 tablet_bootstrap.cc:654] T a49b6f596d724b35b757ee69fe15e5a5 P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:17.229158 14185 tablet_bootstrap.cc:654] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:17.229832 14126 tablet_bootstrap.cc:492] T a49b6f596d724b35b757ee69fe15e5a5 P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:17.229908 14126 ts_tablet_manager.cc:1367] T a49b6f596d724b35b757ee69fe15e5a5 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:17.230188 14130 tablet_bootstrap.cc:492] T c047c61f42f04484842125847ca677aa P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:17.230293 14126 raft_consensus.cc:348] T a49b6f596d724b35b757ee69fe15e5a5 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:17.230396 14126 raft_consensus.cc:374] T a49b6f596d724b35b757ee69fe15e5a5 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:17.230425 14126 raft_consensus.cc:729] T a49b6f596d724b35b757ee69fe15e5a5 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:17.230543 14126 consensus_queue.cc:260] T a49b6f596d724b35b757ee69fe15e5a5 P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:17.230806 14126 ts_tablet_manager.cc:1398] T a49b6f596d724b35b757ee69fe15e5a5 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:17.231034 14130 tablet_bootstrap.cc:654] T c047c61f42f04484842125847ca677aa P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:17.232563 14130 tablet_bootstrap.cc:492] T c047c61f42f04484842125847ca677aa P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:17.232648 14130 ts_tablet_manager.cc:1367] T c047c61f42f04484842125847ca677aa P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:56:17.232923 14127 tablet_bootstrap.cc:492] T 4647faf6a457415d8ae2b6691f452eab P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:17.233009 14127 ts_tablet_manager.cc:1367] T 4647faf6a457415d8ae2b6691f452eab P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240503 13:56:17.233103 14185 tablet_bootstrap.cc:492] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:17.233099 14130 raft_consensus.cc:348] T c047c61f42f04484842125847ca677aa P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:17.233179 14185 ts_tablet_manager.cc:1367] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 13:56:17.233192 14130 raft_consensus.cc:374] T c047c61f42f04484842125847ca677aa P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:17.233222 14130 raft_consensus.cc:729] T c047c61f42f04484842125847ca677aa P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:17.233332 14130 consensus_queue.cc:260] T c047c61f42f04484842125847ca677aa P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:17.233426 14127 raft_consensus.cc:348] T 4647faf6a457415d8ae2b6691f452eab P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:17.233532 14127 raft_consensus.cc:374] T 4647faf6a457415d8ae2b6691f452eab P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:17.233557 14127 raft_consensus.cc:729] T 4647faf6a457415d8ae2b6691f452eab P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:17.233608 14130 ts_tablet_manager.cc:1398] T c047c61f42f04484842125847ca677aa P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:17.233656 14127 consensus_queue.cc:260] T 4647faf6a457415d8ae2b6691f452eab P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:17.233644 14185 raft_consensus.cc:348] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:17.233736 14185 raft_consensus.cc:374] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:17.233768 14185 raft_consensus.cc:729] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:17.233897 14127 ts_tablet_manager.cc:1398] T 4647faf6a457415d8ae2b6691f452eab P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:17.233881 14185 consensus_queue.cc:260] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:17.234128 14185 ts_tablet_manager.cc:1398] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:17.234287 14185 tablet_bootstrap.cc:492] T 4647faf6a457415d8ae2b6691f452eab P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:17.235159 14185 tablet_bootstrap.cc:654] T 4647faf6a457415d8ae2b6691f452eab P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:17.236531 14185 tablet_bootstrap.cc:492] T 4647faf6a457415d8ae2b6691f452eab P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:17.236610 14185 ts_tablet_manager.cc:1367] T 4647faf6a457415d8ae2b6691f452eab P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 13:56:17.237016 14185 raft_consensus.cc:348] T 4647faf6a457415d8ae2b6691f452eab P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:17.237110 14185 raft_consensus.cc:374] T 4647faf6a457415d8ae2b6691f452eab P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:17.237143 14185 raft_consensus.cc:729] T 4647faf6a457415d8ae2b6691f452eab P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:17.237258 14185 consensus_queue.cc:260] T 4647faf6a457415d8ae2b6691f452eab P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:17.237522 14185 ts_tablet_manager.cc:1398] T 4647faf6a457415d8ae2b6691f452eab P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:17.275826 14043 raft_consensus.cc:482] T a49b6f596d724b35b757ee69fe15e5a5 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:17.275951 14043 raft_consensus.cc:504] T a49b6f596d724b35b757ee69fe15e5a5 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:17.276356 14043 leader_election.cc:288] T a49b6f596d724b35b757ee69fe15e5a5 P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279), 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:17.276790 13315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a49b6f596d724b35b757ee69fe15e5a5" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703" is_pre_election: true
I20240503 13:56:17.276819 13054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a49b6f596d724b35b757ee69fe15e5a5" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e" is_pre_election: true
I20240503 13:56:17.276930 13315 raft_consensus.cc:2454] T a49b6f596d724b35b757ee69fe15e5a5 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 0.
I20240503 13:56:17.276957 13054 raft_consensus.cc:2454] T a49b6f596d724b35b757ee69fe15e5a5 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 0.
I20240503 13:56:17.277249 13121 leader_election.cc:302] T a49b6f596d724b35b757ee69fe15e5a5 P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ccf5a4ac450414c9d3579bd5800b770, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:17.277436 14043 raft_consensus.cc:2781] T a49b6f596d724b35b757ee69fe15e5a5 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:17.277498 14043 raft_consensus.cc:482] T a49b6f596d724b35b757ee69fe15e5a5 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:17.277542 14043 raft_consensus.cc:3037] T a49b6f596d724b35b757ee69fe15e5a5 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:17.278308 14043 raft_consensus.cc:504] T a49b6f596d724b35b757ee69fe15e5a5 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:17.278687 14043 leader_election.cc:288] T a49b6f596d724b35b757ee69fe15e5a5 P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 election: Requested vote from peers ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279), 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:17.279012 13054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a49b6f596d724b35b757ee69fe15e5a5" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e"
I20240503 13:56:17.279055 13315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a49b6f596d724b35b757ee69fe15e5a5" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703"
I20240503 13:56:17.279129 13054 raft_consensus.cc:3037] T a49b6f596d724b35b757ee69fe15e5a5 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:17.279167 13315 raft_consensus.cc:3037] T a49b6f596d724b35b757ee69fe15e5a5 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:17.279927 13054 raft_consensus.cc:2454] T a49b6f596d724b35b757ee69fe15e5a5 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 1.
I20240503 13:56:17.279970 13315 raft_consensus.cc:2454] T a49b6f596d724b35b757ee69fe15e5a5 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 1.
I20240503 13:56:17.280200 13119 leader_election.cc:302] T a49b6f596d724b35b757ee69fe15e5a5 P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 8ccf5a4ac450414c9d3579bd5800b770; no voters:
I20240503 13:56:17.280385 14043 raft_consensus.cc:2781] T a49b6f596d724b35b757ee69fe15e5a5 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:17.280479 14043 raft_consensus.cc:686] T a49b6f596d724b35b757ee69fe15e5a5 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 LEADER]: Becoming Leader. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Running, Role: LEADER
I20240503 13:56:17.280663 14043 consensus_queue.cc:237] T a49b6f596d724b35b757ee69fe15e5a5 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:17.282238 12907 catalog_manager.cc:5484] T a49b6f596d724b35b757ee69fe15e5a5 P 8ccf5a4ac450414c9d3579bd5800b770 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66). New cstate: current_term: 1 leader_uuid: "8ccf5a4ac450414c9d3579bd5800b770" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } }
I20240503 13:56:17.312176 14100 raft_consensus.cc:482] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:17.312314 14100 raft_consensus.cc:504] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:17.312763 14100 leader_election.cc:288] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279), 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:17.313194 13315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c047c61f42f04484842125847ca677aa" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703" is_pre_election: true
I20240503 13:56:17.313194 13054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c047c61f42f04484842125847ca677aa" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e" is_pre_election: true
I20240503 13:56:17.313329 13315 raft_consensus.cc:2454] T c047c61f42f04484842125847ca677aa P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 0.
I20240503 13:56:17.313338 13054 raft_consensus.cc:2454] T c047c61f42f04484842125847ca677aa P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 0.
I20240503 13:56:17.313619 13381 leader_election.cc:302] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 7690ee79577c45e3a2c8f8b552c0bff8; no voters:
I20240503 13:56:17.313778 14100 raft_consensus.cc:2781] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:17.313833 14100 raft_consensus.cc:482] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:17.313875 14100 raft_consensus.cc:3037] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:17.314703 14100 raft_consensus.cc:504] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:17.315032 14100 leader_election.cc:288] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 election: Requested vote from peers ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279), 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:17.315366 13315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c047c61f42f04484842125847ca677aa" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703"
I20240503 13:56:17.315379 13054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c047c61f42f04484842125847ca677aa" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e"
I20240503 13:56:17.315477 13315 raft_consensus.cc:3037] T c047c61f42f04484842125847ca677aa P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:17.315485 13054 raft_consensus.cc:3037] T c047c61f42f04484842125847ca677aa P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:17.316152 13315 raft_consensus.cc:2454] T c047c61f42f04484842125847ca677aa P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 1.
I20240503 13:56:17.316228 13054 raft_consensus.cc:2454] T c047c61f42f04484842125847ca677aa P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 1.
I20240503 13:56:17.316449 13383 leader_election.cc:302] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:17.316627 14100 raft_consensus.cc:2781] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:17.316709 14100 raft_consensus.cc:686] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 LEADER]: Becoming Leader. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Running, Role: LEADER
I20240503 13:56:17.316843 14100 consensus_queue.cc:237] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:17.318586 12907 catalog_manager.cc:5484] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68). New cstate: current_term: 1 leader_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } }
I20240503 13:56:17.559548 14053 raft_consensus.cc:482] T 4647faf6a457415d8ae2b6691f452eab P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:17.559708 14053 raft_consensus.cc:504] T 4647faf6a457415d8ae2b6691f452eab P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:17.560220 14053 leader_election.cc:288] T 4647faf6a457415d8ae2b6691f452eab P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387), 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:17.560902 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4647faf6a457415d8ae2b6691f452eab" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
I20240503 13:56:17.561103 13446 raft_consensus.cc:2454] T 4647faf6a457415d8ae2b6691f452eab P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 0.
I20240503 13:56:17.561435 13249 leader_election.cc:302] T 4647faf6a457415d8ae2b6691f452eab P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:17.561625 14053 raft_consensus.cc:2781] T 4647faf6a457415d8ae2b6691f452eab P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:17.561678 14053 raft_consensus.cc:482] T 4647faf6a457415d8ae2b6691f452eab P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:17.561713 14053 raft_consensus.cc:3037] T 4647faf6a457415d8ae2b6691f452eab P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:17.562603 14053 raft_consensus.cc:504] T 4647faf6a457415d8ae2b6691f452eab P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:17.563017 14053 leader_election.cc:288] T 4647faf6a457415d8ae2b6691f452eab P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: Requested vote from peers 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387), 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:17.563382 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4647faf6a457415d8ae2b6691f452eab" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
I20240503 13:56:17.563519 13446 raft_consensus.cc:3037] T 4647faf6a457415d8ae2b6691f452eab P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:17.564167 13184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4647faf6a457415d8ae2b6691f452eab" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770" is_pre_election: true
I20240503 13:56:17.564193 13185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4647faf6a457415d8ae2b6691f452eab" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770"
I20240503 13:56:17.564327 13184 raft_consensus.cc:2454] T 4647faf6a457415d8ae2b6691f452eab P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 0.
I20240503 13:56:17.564548 13446 raft_consensus.cc:2454] T 4647faf6a457415d8ae2b6691f452eab P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 1.
I20240503 13:56:17.564858 13249 leader_election.cc:302] T 4647faf6a457415d8ae2b6691f452eab P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, ba0194098aa84449ae17e8a0d4c43703; no voters: 8ccf5a4ac450414c9d3579bd5800b770
I20240503 13:56:17.565028 14053 raft_consensus.cc:2781] T 4647faf6a457415d8ae2b6691f452eab P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:17.565140 14053 raft_consensus.cc:686] T 4647faf6a457415d8ae2b6691f452eab P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Becoming Leader. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Running, Role: LEADER
I20240503 13:56:17.565279 14053 consensus_queue.cc:237] T 4647faf6a457415d8ae2b6691f452eab P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:17.567034 12907 catalog_manager.cc:5484] T 4647faf6a457415d8ae2b6691f452eab P ba0194098aa84449ae17e8a0d4c43703 reported cstate change: term changed from 0 to 1, leader changed from <none> to ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67). New cstate: current_term: 1 leader_uuid: "ba0194098aa84449ae17e8a0d4c43703" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } }
I20240503 13:56:17.667479 14043 consensus_queue.cc:1035] T a49b6f596d724b35b757ee69fe15e5a5 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:17.670656 14043 consensus_queue.cc:1035] T a49b6f596d724b35b757ee69fe15e5a5 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 13:56:17.706207 14251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:56:17.706524 14251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:56:17.709872 14251 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240503 13:56:17.775794 14100 consensus_queue.cc:1035] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:17.781204 14049 consensus_queue.cc:1035] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:17.806689 13184 raft_consensus.cc:3037] T 4647faf6a457415d8ae2b6691f452eab P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:17.806912 13446 raft_consensus.cc:1261] T 4647faf6a457415d8ae2b6691f452eab P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:17.807765 13184 raft_consensus.cc:1261] T 4647faf6a457415d8ae2b6691f452eab P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:17.808125 14053 consensus_queue.cc:1035] T 4647faf6a457415d8ae2b6691f452eab P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:17.810292 14053 consensus_queue.cc:1035] T 4647faf6a457415d8ae2b6691f452eab P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 103.392 ms
time per row: 0.0516962 ms
I20240503 13:56:17.867707 12909 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:56:17.873837 13164 tablet_service.cc:1467] Processing CreateTablet for tablet b68bfa77367e4297a612610812eb8dfd (DEFAULT_TABLE table=rebalance_test_table_4 [id=6e1f1c4458104a159283843ed95fe485]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:17.874197 13164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b68bfa77367e4297a612610812eb8dfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:17.874410 13165 tablet_service.cc:1467] Processing CreateTablet for tablet b26d8338f20749ed92f5d7670e9549d5 (DEFAULT_TABLE table=rebalance_test_table_4 [id=6e1f1c4458104a159283843ed95fe485]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:17.874410 13425 tablet_service.cc:1467] Processing CreateTablet for tablet b26d8338f20749ed92f5d7670e9549d5 (DEFAULT_TABLE table=rebalance_test_table_4 [id=6e1f1c4458104a159283843ed95fe485]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:17.874675 13425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b26d8338f20749ed92f5d7670e9549d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:17.874675 13165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b26d8338f20749ed92f5d7670e9549d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:17.876032 14311 tablet_bootstrap.cc:492] T b68bfa77367e4297a612610812eb8dfd P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:17.876279 13032 tablet_service.cc:1467] Processing CreateTablet for tablet b68bfa77367e4297a612610812eb8dfd (DEFAULT_TABLE table=rebalance_test_table_4 [id=6e1f1c4458104a159283843ed95fe485]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:17.876603 13032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b68bfa77367e4297a612610812eb8dfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:17.876856 14312 tablet_bootstrap.cc:492] T b26d8338f20749ed92f5d7670e9549d5 P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:17.877187 14311 tablet_bootstrap.cc:654] T b68bfa77367e4297a612610812eb8dfd P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:17.877147 13033 tablet_service.cc:1467] Processing CreateTablet for tablet 809936cd39ce41229cc779f59c69900a (DEFAULT_TABLE table=rebalance_test_table_4 [id=6e1f1c4458104a159283843ed95fe485]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:17.877365 13033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 809936cd39ce41229cc779f59c69900a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:17.877715 14312 tablet_bootstrap.cc:654] T b26d8338f20749ed92f5d7670e9549d5 P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:17.878756 14315 tablet_bootstrap.cc:492] T b68bfa77367e4297a612610812eb8dfd P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:17.878808 14312 tablet_bootstrap.cc:492] T b26d8338f20749ed92f5d7670e9549d5 P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:17.878904 14312 ts_tablet_manager.cc:1367] T b26d8338f20749ed92f5d7670e9549d5 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:17.878990 13034 tablet_service.cc:1467] Processing CreateTablet for tablet b26d8338f20749ed92f5d7670e9549d5 (DEFAULT_TABLE table=rebalance_test_table_4 [id=6e1f1c4458104a159283843ed95fe485]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:17.879200 13034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b26d8338f20749ed92f5d7670e9549d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:17.879361 14312 raft_consensus.cc:348] T b26d8338f20749ed92f5d7670e9549d5 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:17.879485 14312 raft_consensus.cc:374] T b26d8338f20749ed92f5d7670e9549d5 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:17.879524 14312 raft_consensus.cc:729] T b26d8338f20749ed92f5d7670e9549d5 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:17.879675 14312 consensus_queue.cc:260] T b26d8338f20749ed92f5d7670e9549d5 P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:17.880048 14315 tablet_bootstrap.cc:654] T b68bfa77367e4297a612610812eb8dfd P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:17.880065 14312 ts_tablet_manager.cc:1398] T b26d8338f20749ed92f5d7670e9549d5 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:17.880213 13425 tablet_service.cc:1467] Processing CreateTablet for tablet 809936cd39ce41229cc779f59c69900a (DEFAULT_TABLE table=rebalance_test_table_4 [id=6e1f1c4458104a159283843ed95fe485]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:17.880421 13425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 809936cd39ce41229cc779f59c69900a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:17.881397 14315 tablet_bootstrap.cc:492] T b68bfa77367e4297a612610812eb8dfd P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:17.881480 14315 ts_tablet_manager.cc:1367] T b68bfa77367e4297a612610812eb8dfd P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:56:17.881424 13294 tablet_service.cc:1467] Processing CreateTablet for tablet b68bfa77367e4297a612610812eb8dfd (DEFAULT_TABLE table=rebalance_test_table_4 [id=6e1f1c4458104a159283843ed95fe485]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:17.881695 13294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b68bfa77367e4297a612610812eb8dfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:17.881886 14315 raft_consensus.cc:348] T b68bfa77367e4297a612610812eb8dfd P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:17.881975 14315 raft_consensus.cc:374] T b68bfa77367e4297a612610812eb8dfd P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:17.882005 14315 raft_consensus.cc:729] T b68bfa77367e4297a612610812eb8dfd P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:17.882117 14315 consensus_queue.cc:260] T b68bfa77367e4297a612610812eb8dfd P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:17.882354 14315 ts_tablet_manager.cc:1398] T b68bfa77367e4297a612610812eb8dfd P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:17.882472 14312 tablet_bootstrap.cc:492] T 809936cd39ce41229cc779f59c69900a P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:17.882540 14315 tablet_bootstrap.cc:492] T 809936cd39ce41229cc779f59c69900a P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:17.883342 14312 tablet_bootstrap.cc:654] T 809936cd39ce41229cc779f59c69900a P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:17.883347 14315 tablet_bootstrap.cc:654] T 809936cd39ce41229cc779f59c69900a P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:17.883975 14317 tablet_bootstrap.cc:492] T b68bfa77367e4297a612610812eb8dfd P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:17.884057 13295 tablet_service.cc:1467] Processing CreateTablet for tablet 809936cd39ce41229cc779f59c69900a (DEFAULT_TABLE table=rebalance_test_table_4 [id=6e1f1c4458104a159283843ed95fe485]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:17.884284 13295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 809936cd39ce41229cc779f59c69900a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:17.884800 14317 tablet_bootstrap.cc:654] T b68bfa77367e4297a612610812eb8dfd P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:17.885147 14315 tablet_bootstrap.cc:492] T 809936cd39ce41229cc779f59c69900a P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:17.885290 14315 ts_tablet_manager.cc:1367] T 809936cd39ce41229cc779f59c69900a P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:56:17.885775 14315 raft_consensus.cc:348] T 809936cd39ce41229cc779f59c69900a P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:17.885864 14315 raft_consensus.cc:374] T 809936cd39ce41229cc779f59c69900a P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:17.885890 14315 raft_consensus.cc:729] T 809936cd39ce41229cc779f59c69900a P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:17.886031 14315 consensus_queue.cc:260] T 809936cd39ce41229cc779f59c69900a P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:17.886348 14315 ts_tablet_manager.cc:1398] T 809936cd39ce41229cc779f59c69900a P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:17.886729 14315 tablet_bootstrap.cc:492] T b26d8338f20749ed92f5d7670e9549d5 P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:17.887401 14311 tablet_bootstrap.cc:492] T b68bfa77367e4297a612610812eb8dfd P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:17.887490 14311 ts_tablet_manager.cc:1367] T b68bfa77367e4297a612610812eb8dfd P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.000s
I20240503 13:56:17.887549 14315 tablet_bootstrap.cc:654] T b26d8338f20749ed92f5d7670e9549d5 P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:17.887867 14312 tablet_bootstrap.cc:492] T 809936cd39ce41229cc779f59c69900a P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:17.887950 14312 ts_tablet_manager.cc:1367] T 809936cd39ce41229cc779f59c69900a P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 13:56:17.887926 14311 raft_consensus.cc:348] T b68bfa77367e4297a612610812eb8dfd P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:17.888029 14311 raft_consensus.cc:374] T b68bfa77367e4297a612610812eb8dfd P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:17.888058 14311 raft_consensus.cc:729] T b68bfa77367e4297a612610812eb8dfd P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:17.888175 14311 consensus_queue.cc:260] T b68bfa77367e4297a612610812eb8dfd P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:17.888365 14312 raft_consensus.cc:348] T 809936cd39ce41229cc779f59c69900a P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:17.888461 14311 ts_tablet_manager.cc:1398] T b68bfa77367e4297a612610812eb8dfd P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:17.888470 14312 raft_consensus.cc:374] T 809936cd39ce41229cc779f59c69900a P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:17.888547 14312 raft_consensus.cc:729] T 809936cd39ce41229cc779f59c69900a P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:17.888650 14311 tablet_bootstrap.cc:492] T b26d8338f20749ed92f5d7670e9549d5 P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:17.888664 14312 consensus_queue.cc:260] T 809936cd39ce41229cc779f59c69900a P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:17.888774 14315 tablet_bootstrap.cc:492] T b26d8338f20749ed92f5d7670e9549d5 P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:17.888837 14315 ts_tablet_manager.cc:1367] T b26d8338f20749ed92f5d7670e9549d5 P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:56:17.888926 14312 ts_tablet_manager.cc:1398] T 809936cd39ce41229cc779f59c69900a P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:17.889223 14315 raft_consensus.cc:348] T b26d8338f20749ed92f5d7670e9549d5 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:17.889308 14315 raft_consensus.cc:374] T b26d8338f20749ed92f5d7670e9549d5 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:17.889336 14315 raft_consensus.cc:729] T b26d8338f20749ed92f5d7670e9549d5 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:17.889442 14315 consensus_queue.cc:260] T b26d8338f20749ed92f5d7670e9549d5 P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:17.889510 14311 tablet_bootstrap.cc:654] T b26d8338f20749ed92f5d7670e9549d5 P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:17.886986 14317 tablet_bootstrap.cc:492] T b68bfa77367e4297a612610812eb8dfd P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:17.889619 14317 ts_tablet_manager.cc:1367] T b68bfa77367e4297a612610812eb8dfd P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 13:56:17.889734 14315 ts_tablet_manager.cc:1398] T b26d8338f20749ed92f5d7670e9549d5 P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:17.890072 14317 raft_consensus.cc:348] T b68bfa77367e4297a612610812eb8dfd P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:17.890188 14317 raft_consensus.cc:374] T b68bfa77367e4297a612610812eb8dfd P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:17.890218 14317 raft_consensus.cc:729] T b68bfa77367e4297a612610812eb8dfd P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:17.890403 14317 consensus_queue.cc:260] T b68bfa77367e4297a612610812eb8dfd P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:17.890674 14317 ts_tablet_manager.cc:1398] T b68bfa77367e4297a612610812eb8dfd P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:17.890815 14317 tablet_bootstrap.cc:492] T 809936cd39ce41229cc779f59c69900a P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:17.891678 14317 tablet_bootstrap.cc:654] T 809936cd39ce41229cc779f59c69900a P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:17.892588 14311 tablet_bootstrap.cc:492] T b26d8338f20749ed92f5d7670e9549d5 P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:17.892664 14311 ts_tablet_manager.cc:1367] T b26d8338f20749ed92f5d7670e9549d5 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:56:17.892712 14317 tablet_bootstrap.cc:492] T 809936cd39ce41229cc779f59c69900a P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:17.892782 14317 ts_tablet_manager.cc:1367] T 809936cd39ce41229cc779f59c69900a P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:56:17.893105 14311 raft_consensus.cc:348] T b26d8338f20749ed92f5d7670e9549d5 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:17.893213 14311 raft_consensus.cc:374] T b26d8338f20749ed92f5d7670e9549d5 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:17.893169 14317 raft_consensus.cc:348] T 809936cd39ce41229cc779f59c69900a P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:17.893241 14311 raft_consensus.cc:729] T b26d8338f20749ed92f5d7670e9549d5 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:17.893273 14317 raft_consensus.cc:374] T 809936cd39ce41229cc779f59c69900a P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:17.893301 14317 raft_consensus.cc:729] T 809936cd39ce41229cc779f59c69900a P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:17.893354 14311 consensus_queue.cc:260] T b26d8338f20749ed92f5d7670e9549d5 P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:17.893405 14317 consensus_queue.cc:260] T 809936cd39ce41229cc779f59c69900a P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:17.893625 14311 ts_tablet_manager.cc:1398] T b26d8338f20749ed92f5d7670e9549d5 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:17.893654 14317 ts_tablet_manager.cc:1398] T 809936cd39ce41229cc779f59c69900a P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:17.965348 14053 raft_consensus.cc:482] T b68bfa77367e4297a612610812eb8dfd P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:17.965490 14053 raft_consensus.cc:504] T b68bfa77367e4297a612610812eb8dfd P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:17.965889 14053 leader_election.cc:288] T b68bfa77367e4297a612610812eb8dfd P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387), 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:17.966348 13054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b68bfa77367e4297a612610812eb8dfd" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e" is_pre_election: true
I20240503 13:56:17.966348 13184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b68bfa77367e4297a612610812eb8dfd" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770" is_pre_election: true
I20240503 13:56:17.966518 13054 raft_consensus.cc:2454] T b68bfa77367e4297a612610812eb8dfd P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 0.
I20240503 13:56:17.966521 13184 raft_consensus.cc:2454] T b68bfa77367e4297a612610812eb8dfd P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 0.
I20240503 13:56:17.966879 13252 leader_election.cc:302] T b68bfa77367e4297a612610812eb8dfd P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ccf5a4ac450414c9d3579bd5800b770, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:17.967089 14053 raft_consensus.cc:2781] T b68bfa77367e4297a612610812eb8dfd P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:17.967156 14053 raft_consensus.cc:482] T b68bfa77367e4297a612610812eb8dfd P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:17.967191 14053 raft_consensus.cc:3037] T b68bfa77367e4297a612610812eb8dfd P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:17.968091 14053 raft_consensus.cc:504] T b68bfa77367e4297a612610812eb8dfd P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:17.968441 14053 leader_election.cc:288] T b68bfa77367e4297a612610812eb8dfd P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: Requested vote from peers 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387), 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:17.968879 13184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b68bfa77367e4297a612610812eb8dfd" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770"
I20240503 13:56:17.968879 13054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b68bfa77367e4297a612610812eb8dfd" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e"
I20240503 13:56:17.969017 13184 raft_consensus.cc:3037] T b68bfa77367e4297a612610812eb8dfd P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:17.969022 13054 raft_consensus.cc:3037] T b68bfa77367e4297a612610812eb8dfd P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:17.970010 13054 raft_consensus.cc:2454] T b68bfa77367e4297a612610812eb8dfd P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 1.
I20240503 13:56:17.970010 13184 raft_consensus.cc:2454] T b68bfa77367e4297a612610812eb8dfd P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 1.
I20240503 13:56:17.970343 13252 leader_election.cc:302] T b68bfa77367e4297a612610812eb8dfd P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ccf5a4ac450414c9d3579bd5800b770, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:17.970606 14053 raft_consensus.cc:2781] T b68bfa77367e4297a612610812eb8dfd P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:17.970718 14053 raft_consensus.cc:686] T b68bfa77367e4297a612610812eb8dfd P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Becoming Leader. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Running, Role: LEADER
I20240503 13:56:17.970863 14053 consensus_queue.cc:237] T b68bfa77367e4297a612610812eb8dfd P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:17.972597 12905 catalog_manager.cc:5484] T b68bfa77367e4297a612610812eb8dfd P ba0194098aa84449ae17e8a0d4c43703 reported cstate change: term changed from 0 to 1, leader changed from <none> to ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67). New cstate: current_term: 1 leader_uuid: "ba0194098aa84449ae17e8a0d4c43703" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:18.075033 14053 raft_consensus.cc:482] T 809936cd39ce41229cc779f59c69900a P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:18.075152 14053 raft_consensus.cc:504] T 809936cd39ce41229cc779f59c69900a P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:18.075565 14053 leader_election.cc:288] T 809936cd39ce41229cc779f59c69900a P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:18.075932 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809936cd39ce41229cc779f59c69900a" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
I20240503 13:56:18.076092 13446 raft_consensus.cc:2454] T 809936cd39ce41229cc779f59c69900a P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 0.
I20240503 13:56:18.076109 13054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809936cd39ce41229cc779f59c69900a" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e" is_pre_election: true
I20240503 13:56:18.076246 13054 raft_consensus.cc:2454] T 809936cd39ce41229cc779f59c69900a P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 0.
I20240503 13:56:18.076421 13249 leader_election.cc:302] T 809936cd39ce41229cc779f59c69900a P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:18.076673 14053 raft_consensus.cc:2781] T 809936cd39ce41229cc779f59c69900a P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:18.076727 14053 raft_consensus.cc:482] T 809936cd39ce41229cc779f59c69900a P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:18.076761 14053 raft_consensus.cc:3037] T 809936cd39ce41229cc779f59c69900a P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:18.077548 14053 raft_consensus.cc:504] T 809936cd39ce41229cc779f59c69900a P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:18.077875 14053 leader_election.cc:288] T 809936cd39ce41229cc779f59c69900a P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: Requested vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:18.078217 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809936cd39ce41229cc779f59c69900a" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
I20240503 13:56:18.078346 13446 raft_consensus.cc:3037] T 809936cd39ce41229cc779f59c69900a P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:18.078318 13054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809936cd39ce41229cc779f59c69900a" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e"
I20240503 13:56:18.078457 13054 raft_consensus.cc:3037] T 809936cd39ce41229cc779f59c69900a P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:18.079099 13446 raft_consensus.cc:2454] T 809936cd39ce41229cc779f59c69900a P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 1.
I20240503 13:56:18.079270 13054 raft_consensus.cc:2454] T 809936cd39ce41229cc779f59c69900a P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 1.
I20240503 13:56:18.079407 13249 leader_election.cc:302] T 809936cd39ce41229cc779f59c69900a P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:18.079582 14053 raft_consensus.cc:2781] T 809936cd39ce41229cc779f59c69900a P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:18.079667 14053 raft_consensus.cc:686] T 809936cd39ce41229cc779f59c69900a P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Becoming Leader. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Running, Role: LEADER
I20240503 13:56:18.079804 14053 consensus_queue.cc:237] T 809936cd39ce41229cc779f59c69900a P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:18.081568 12905 catalog_manager.cc:5484] T 809936cd39ce41229cc779f59c69900a P ba0194098aa84449ae17e8a0d4c43703 reported cstate change: term changed from 0 to 1, leader changed from <none> to ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67). New cstate: current_term: 1 leader_uuid: "ba0194098aa84449ae17e8a0d4c43703" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } }
I20240503 13:56:18.131352 14141 raft_consensus.cc:482] T b26d8338f20749ed92f5d7670e9549d5 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:18.131494 14141 raft_consensus.cc:504] T b26d8338f20749ed92f5d7670e9549d5 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:18.131940 14141 leader_election.cc:288] T b26d8338f20749ed92f5d7670e9549d5 P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:18.132411 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b26d8338f20749ed92f5d7670e9549d5" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
I20240503 13:56:18.132516 13054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b26d8338f20749ed92f5d7670e9549d5" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e" is_pre_election: true
I20240503 13:56:18.132596 13446 raft_consensus.cc:2454] T b26d8338f20749ed92f5d7670e9549d5 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 0.
I20240503 13:56:18.132658 13054 raft_consensus.cc:2454] T b26d8338f20749ed92f5d7670e9549d5 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 0.
I20240503 13:56:18.132886 13118 leader_election.cc:302] T b26d8338f20749ed92f5d7670e9549d5 P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, 8ccf5a4ac450414c9d3579bd5800b770; no voters:
I20240503 13:56:18.133059 14141 raft_consensus.cc:2781] T b26d8338f20749ed92f5d7670e9549d5 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:18.133119 14141 raft_consensus.cc:482] T b26d8338f20749ed92f5d7670e9549d5 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:18.133164 14141 raft_consensus.cc:3037] T b26d8338f20749ed92f5d7670e9549d5 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:18.133987 14141 raft_consensus.cc:504] T b26d8338f20749ed92f5d7670e9549d5 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:18.134343 14141 leader_election.cc:288] T b26d8338f20749ed92f5d7670e9549d5 P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 election: Requested vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:18.134706 13054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b26d8338f20749ed92f5d7670e9549d5" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e"
I20240503 13:56:18.134727 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b26d8338f20749ed92f5d7670e9549d5" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
I20240503 13:56:18.134820 13446 raft_consensus.cc:3037] T b26d8338f20749ed92f5d7670e9549d5 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:18.134819 13054 raft_consensus.cc:3037] T b26d8338f20749ed92f5d7670e9549d5 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:18.135520 13446 raft_consensus.cc:2454] T b26d8338f20749ed92f5d7670e9549d5 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 1.
I20240503 13:56:18.135566 13054 raft_consensus.cc:2454] T b26d8338f20749ed92f5d7670e9549d5 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 1.
I20240503 13:56:18.135788 13118 leader_election.cc:302] T b26d8338f20749ed92f5d7670e9549d5 P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, 8ccf5a4ac450414c9d3579bd5800b770; no voters:
I20240503 13:56:18.135955 14141 raft_consensus.cc:2781] T b26d8338f20749ed92f5d7670e9549d5 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:18.136042 14141 raft_consensus.cc:686] T b26d8338f20749ed92f5d7670e9549d5 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 LEADER]: Becoming Leader. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Running, Role: LEADER
I20240503 13:56:18.136173 14141 consensus_queue.cc:237] T b26d8338f20749ed92f5d7670e9549d5 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:18.137909 12905 catalog_manager.cc:5484] T b26d8338f20749ed92f5d7670e9549d5 P 8ccf5a4ac450414c9d3579bd5800b770 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66). New cstate: current_term: 1 leader_uuid: "8ccf5a4ac450414c9d3579bd5800b770" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } }
W20240503 13:56:18.287714 14327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:56:18.288000 14327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:56:18.291275 14327 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240503 13:56:18.354735 14053 consensus_queue.cc:1035] T b68bfa77367e4297a612610812eb8dfd P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:18.358844 14113 consensus_queue.cc:1035] T b68bfa77367e4297a612610812eb8dfd P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:18.375634 13446 raft_consensus.cc:1261] T 809936cd39ce41229cc779f59c69900a P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:18.376083 14365 consensus_queue.cc:1035] T 809936cd39ce41229cc779f59c69900a P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:18.378280 13053 raft_consensus.cc:1261] T 809936cd39ce41229cc779f59c69900a P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:18.378916 13053 raft_consensus.cc:1261] T b26d8338f20749ed92f5d7670e9549d5 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Refusing update from remote peer 8ccf5a4ac450414c9d3579bd5800b770: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:18.379748 13447 raft_consensus.cc:1261] T b26d8338f20749ed92f5d7670e9549d5 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer 8ccf5a4ac450414c9d3579bd5800b770: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:18.380373 14253 consensus_queue.cc:1035] T b26d8338f20749ed92f5d7670e9549d5 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:18.381170 14253 consensus_queue.cc:1035] T b26d8338f20749ed92f5d7670e9549d5 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:18.381534 14053 consensus_queue.cc:1035] T 809936cd39ce41229cc779f59c69900a P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 104.018 ms
time per row: 0.052009 ms
I20240503 13:56:18.439280 12904 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:56:18.444674 13295 tablet_service.cc:1467] Processing CreateTablet for tablet bb4f0bb47a29437090750c4486df2f02 (DEFAULT_TABLE table=rebalance_test_table_5 [id=ef32d41e01cb46ed894c4ff29e5c0e52]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:18.444726 13165 tablet_service.cc:1467] Processing CreateTablet for tablet bb4f0bb47a29437090750c4486df2f02 (DEFAULT_TABLE table=rebalance_test_table_5 [id=ef32d41e01cb46ed894c4ff29e5c0e52]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:18.444877 13425 tablet_service.cc:1467] Processing CreateTablet for tablet bb4f0bb47a29437090750c4486df2f02 (DEFAULT_TABLE table=rebalance_test_table_5 [id=ef32d41e01cb46ed894c4ff29e5c0e52]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:18.445050 13165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb4f0bb47a29437090750c4486df2f02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:18.445050 13295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb4f0bb47a29437090750c4486df2f02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:18.445112 13425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb4f0bb47a29437090750c4486df2f02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:18.446805 14390 tablet_bootstrap.cc:492] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:18.447299 14391 tablet_bootstrap.cc:492] T bb4f0bb47a29437090750c4486df2f02 P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:18.448004 14390 tablet_bootstrap.cc:654] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:18.448043 13165 tablet_service.cc:1467] Processing CreateTablet for tablet 9eb0d911701b4b2bb33a6b7ab59f08aa (DEFAULT_TABLE table=rebalance_test_table_5 [id=ef32d41e01cb46ed894c4ff29e5c0e52]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:18.448189 14391 tablet_bootstrap.cc:654] T bb4f0bb47a29437090750c4486df2f02 P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:18.448280 13165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9eb0d911701b4b2bb33a6b7ab59f08aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:18.448602 13034 tablet_service.cc:1467] Processing CreateTablet for tablet b47b28a4766a425e8b9bdfea882f979d (DEFAULT_TABLE table=rebalance_test_table_5 [id=ef32d41e01cb46ed894c4ff29e5c0e52]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:18.448865 13034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b47b28a4766a425e8b9bdfea882f979d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:18.448915 13164 tablet_service.cc:1467] Processing CreateTablet for tablet b47b28a4766a425e8b9bdfea882f979d (DEFAULT_TABLE table=rebalance_test_table_5 [id=ef32d41e01cb46ed894c4ff29e5c0e52]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:18.449136 13164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b47b28a4766a425e8b9bdfea882f979d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:18.450611 14391 tablet_bootstrap.cc:492] T bb4f0bb47a29437090750c4486df2f02 P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:18.450685 14391 ts_tablet_manager.cc:1367] T bb4f0bb47a29437090750c4486df2f02 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:56:18.450995 13425 tablet_service.cc:1467] Processing CreateTablet for tablet 9eb0d911701b4b2bb33a6b7ab59f08aa (DEFAULT_TABLE table=rebalance_test_table_5 [id=ef32d41e01cb46ed894c4ff29e5c0e52]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:18.451246 13425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9eb0d911701b4b2bb33a6b7ab59f08aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:18.451294 14391 raft_consensus.cc:348] T bb4f0bb47a29437090750c4486df2f02 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:18.451396 14391 raft_consensus.cc:374] T bb4f0bb47a29437090750c4486df2f02 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:18.451426 14391 raft_consensus.cc:729] T bb4f0bb47a29437090750c4486df2f02 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:18.451529 13295 tablet_service.cc:1467] Processing CreateTablet for tablet 9eb0d911701b4b2bb33a6b7ab59f08aa (DEFAULT_TABLE table=rebalance_test_table_5 [id=ef32d41e01cb46ed894c4ff29e5c0e52]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:18.451553 13294 tablet_service.cc:1467] Processing CreateTablet for tablet b47b28a4766a425e8b9bdfea882f979d (DEFAULT_TABLE table=rebalance_test_table_5 [id=ef32d41e01cb46ed894c4ff29e5c0e52]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:18.451586 14391 consensus_queue.cc:260] T bb4f0bb47a29437090750c4486df2f02 P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:18.451776 13295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9eb0d911701b4b2bb33a6b7ab59f08aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:18.451860 14391 ts_tablet_manager.cc:1398] T bb4f0bb47a29437090750c4486df2f02 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:18.452026 14391 tablet_bootstrap.cc:492] T 9eb0d911701b4b2bb33a6b7ab59f08aa P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:18.452353 13294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b47b28a4766a425e8b9bdfea882f979d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:18.452899 14391 tablet_bootstrap.cc:654] T 9eb0d911701b4b2bb33a6b7ab59f08aa P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:18.453493 14392 tablet_bootstrap.cc:492] T bb4f0bb47a29437090750c4486df2f02 P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:18.454042 14391 tablet_bootstrap.cc:492] T 9eb0d911701b4b2bb33a6b7ab59f08aa P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:18.454116 14391 ts_tablet_manager.cc:1367] T 9eb0d911701b4b2bb33a6b7ab59f08aa P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:56:18.454347 14392 tablet_bootstrap.cc:654] T bb4f0bb47a29437090750c4486df2f02 P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:18.454558 14391 raft_consensus.cc:348] T 9eb0d911701b4b2bb33a6b7ab59f08aa P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:18.454680 14391 raft_consensus.cc:374] T 9eb0d911701b4b2bb33a6b7ab59f08aa P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:18.454715 14391 raft_consensus.cc:729] T 9eb0d911701b4b2bb33a6b7ab59f08aa P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:18.454829 14391 consensus_queue.cc:260] T 9eb0d911701b4b2bb33a6b7ab59f08aa P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:18.455111 14391 ts_tablet_manager.cc:1398] T 9eb0d911701b4b2bb33a6b7ab59f08aa P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:18.455273 14391 tablet_bootstrap.cc:492] T b47b28a4766a425e8b9bdfea882f979d P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:18.455349 14395 tablet_bootstrap.cc:492] T b47b28a4766a425e8b9bdfea882f979d P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:18.455399 14392 tablet_bootstrap.cc:492] T bb4f0bb47a29437090750c4486df2f02 P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:18.455502 14392 ts_tablet_manager.cc:1367] T bb4f0bb47a29437090750c4486df2f02 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:56:18.455907 14392 raft_consensus.cc:348] T bb4f0bb47a29437090750c4486df2f02 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:18.456007 14392 raft_consensus.cc:374] T bb4f0bb47a29437090750c4486df2f02 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:18.456036 14392 raft_consensus.cc:729] T bb4f0bb47a29437090750c4486df2f02 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:18.456086 14391 tablet_bootstrap.cc:654] T b47b28a4766a425e8b9bdfea882f979d P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:18.456146 14392 consensus_queue.cc:260] T bb4f0bb47a29437090750c4486df2f02 P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:18.456419 14392 ts_tablet_manager.cc:1398] T bb4f0bb47a29437090750c4486df2f02 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:18.456770 14392 tablet_bootstrap.cc:492] T 9eb0d911701b4b2bb33a6b7ab59f08aa P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:18.457175 14395 tablet_bootstrap.cc:654] T b47b28a4766a425e8b9bdfea882f979d P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:18.457727 14392 tablet_bootstrap.cc:654] T 9eb0d911701b4b2bb33a6b7ab59f08aa P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:18.459022 14391 tablet_bootstrap.cc:492] T b47b28a4766a425e8b9bdfea882f979d P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:18.459023 14390 tablet_bootstrap.cc:492] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:18.459105 14391 ts_tablet_manager.cc:1367] T b47b28a4766a425e8b9bdfea882f979d P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:56:18.459108 14390 ts_tablet_manager.cc:1367] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 13:56:18.459488 14392 tablet_bootstrap.cc:492] T 9eb0d911701b4b2bb33a6b7ab59f08aa P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:18.459546 14392 ts_tablet_manager.cc:1367] T 9eb0d911701b4b2bb33a6b7ab59f08aa P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:56:18.459520 14390 raft_consensus.cc:348] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:18.459520 14391 raft_consensus.cc:348] T b47b28a4766a425e8b9bdfea882f979d P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:18.459619 14391 raft_consensus.cc:374] T b47b28a4766a425e8b9bdfea882f979d P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:18.459619 14390 raft_consensus.cc:374] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:18.459651 14391 raft_consensus.cc:729] T b47b28a4766a425e8b9bdfea882f979d P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:18.459651 14390 raft_consensus.cc:729] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:18.459771 14390 consensus_queue.cc:260] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:18.459771 14391 consensus_queue.cc:260] T b47b28a4766a425e8b9bdfea882f979d P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:18.459904 14392 raft_consensus.cc:348] T 9eb0d911701b4b2bb33a6b7ab59f08aa P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:18.460012 14392 raft_consensus.cc:374] T 9eb0d911701b4b2bb33a6b7ab59f08aa P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:18.460044 14392 raft_consensus.cc:729] T 9eb0d911701b4b2bb33a6b7ab59f08aa P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:18.460052 14390 ts_tablet_manager.cc:1398] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:18.460052 14391 ts_tablet_manager.cc:1398] T b47b28a4766a425e8b9bdfea882f979d P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:18.460208 14390 tablet_bootstrap.cc:492] T 9eb0d911701b4b2bb33a6b7ab59f08aa P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:18.460180 14392 consensus_queue.cc:260] T 9eb0d911701b4b2bb33a6b7ab59f08aa P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:18.461086 14390 tablet_bootstrap.cc:654] T 9eb0d911701b4b2bb33a6b7ab59f08aa P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:18.461611 14395 tablet_bootstrap.cc:492] T b47b28a4766a425e8b9bdfea882f979d P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:18.461752 14395 ts_tablet_manager.cc:1367] T b47b28a4766a425e8b9bdfea882f979d P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 13:56:18.462236 14395 raft_consensus.cc:348] T b47b28a4766a425e8b9bdfea882f979d P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:18.462438 14395 raft_consensus.cc:374] T b47b28a4766a425e8b9bdfea882f979d P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:18.462481 14395 raft_consensus.cc:729] T b47b28a4766a425e8b9bdfea882f979d P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:18.462633 14392 ts_tablet_manager.cc:1398] T 9eb0d911701b4b2bb33a6b7ab59f08aa P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:56:18.462606 14395 consensus_queue.cc:260] T b47b28a4766a425e8b9bdfea882f979d P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:18.463325 14390 tablet_bootstrap.cc:492] T 9eb0d911701b4b2bb33a6b7ab59f08aa P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:18.463407 14390 ts_tablet_manager.cc:1367] T 9eb0d911701b4b2bb33a6b7ab59f08aa P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:56:18.463449 14395 ts_tablet_manager.cc:1398] T b47b28a4766a425e8b9bdfea882f979d P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:18.463776 14390 raft_consensus.cc:348] T 9eb0d911701b4b2bb33a6b7ab59f08aa P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:18.463874 14390 raft_consensus.cc:374] T 9eb0d911701b4b2bb33a6b7ab59f08aa P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:18.463901 14390 raft_consensus.cc:729] T 9eb0d911701b4b2bb33a6b7ab59f08aa P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:18.464013 14390 consensus_queue.cc:260] T 9eb0d911701b4b2bb33a6b7ab59f08aa P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:18.464267 14390 ts_tablet_manager.cc:1398] T 9eb0d911701b4b2bb33a6b7ab59f08aa P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:18.464419 14390 tablet_bootstrap.cc:492] T b47b28a4766a425e8b9bdfea882f979d P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:18.465348 14390 tablet_bootstrap.cc:654] T b47b28a4766a425e8b9bdfea882f979d P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:18.466480 14390 tablet_bootstrap.cc:492] T b47b28a4766a425e8b9bdfea882f979d P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:18.466594 14390 ts_tablet_manager.cc:1367] T b47b28a4766a425e8b9bdfea882f979d P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:56:18.467038 14390 raft_consensus.cc:348] T b47b28a4766a425e8b9bdfea882f979d P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:18.467154 14390 raft_consensus.cc:374] T b47b28a4766a425e8b9bdfea882f979d P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:18.467190 14390 raft_consensus.cc:729] T b47b28a4766a425e8b9bdfea882f979d P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:18.467289 14390 consensus_queue.cc:260] T b47b28a4766a425e8b9bdfea882f979d P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:18.467646 14390 ts_tablet_manager.cc:1398] T b47b28a4766a425e8b9bdfea882f979d P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:18.655064 14365 raft_consensus.cc:482] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:18.655220 14365 raft_consensus.cc:504] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:18.655611 14365 leader_election.cc:288] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387), 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:18.656055 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb4f0bb47a29437090750c4486df2f02" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
I20240503 13:56:18.656050 13184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb4f0bb47a29437090750c4486df2f02" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770" is_pre_election: true
I20240503 13:56:18.656175 13184 raft_consensus.cc:2454] T bb4f0bb47a29437090750c4486df2f02 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 0.
I20240503 13:56:18.656186 13447 raft_consensus.cc:2454] T bb4f0bb47a29437090750c4486df2f02 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 0.
I20240503 13:56:18.656478 13252 leader_election.cc:302] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ccf5a4ac450414c9d3579bd5800b770, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:18.656674 14365 raft_consensus.cc:2781] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:18.656728 14365 raft_consensus.cc:482] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:18.656752 14365 raft_consensus.cc:3037] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:18.657971 14365 raft_consensus.cc:504] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:18.658401 14365 leader_election.cc:288] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: Requested vote from peers 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387), 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:18.658638 14365 raft_consensus.cc:482] T b47b28a4766a425e8b9bdfea882f979d P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:18.658711 14365 raft_consensus.cc:504] T b47b28a4766a425e8b9bdfea882f979d P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:18.658749 13184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb4f0bb47a29437090750c4486df2f02" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770"
I20240503 13:56:18.658847 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb4f0bb47a29437090750c4486df2f02" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
I20240503 13:56:18.658898 13184 raft_consensus.cc:3037] T bb4f0bb47a29437090750c4486df2f02 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:18.658944 13447 raft_consensus.cc:3037] T bb4f0bb47a29437090750c4486df2f02 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:18.659031 14365 leader_election.cc:288] T b47b28a4766a425e8b9bdfea882f979d P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:18.659392 13052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b47b28a4766a425e8b9bdfea882f979d" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e" is_pre_election: true
I20240503 13:56:18.659523 13052 raft_consensus.cc:2454] T b47b28a4766a425e8b9bdfea882f979d P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 0.
I20240503 13:56:18.659786 13250 leader_election.cc:302] T b47b28a4766a425e8b9bdfea882f979d P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:18.659847 13447 raft_consensus.cc:2454] T bb4f0bb47a29437090750c4486df2f02 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 1.
I20240503 13:56:18.659847 13184 raft_consensus.cc:2454] T bb4f0bb47a29437090750c4486df2f02 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 1.
I20240503 13:56:18.659938 14365 raft_consensus.cc:2781] T b47b28a4766a425e8b9bdfea882f979d P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:18.659986 14365 raft_consensus.cc:482] T b47b28a4766a425e8b9bdfea882f979d P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:18.660020 14365 raft_consensus.cc:3037] T b47b28a4766a425e8b9bdfea882f979d P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:18.660125 13252 leader_election.cc:302] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ccf5a4ac450414c9d3579bd5800b770, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:18.660269 13184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b47b28a4766a425e8b9bdfea882f979d" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770" is_pre_election: true
I20240503 13:56:18.660292 14113 raft_consensus.cc:2781] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:18.660368 13184 raft_consensus.cc:2454] T b47b28a4766a425e8b9bdfea882f979d P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 0.
I20240503 13:56:18.660372 14113 raft_consensus.cc:686] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Becoming Leader. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Running, Role: LEADER
I20240503 13:56:18.660612 14113 consensus_queue.cc:237] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:18.660938 14365 raft_consensus.cc:504] T b47b28a4766a425e8b9bdfea882f979d P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:18.661228 14365 leader_election.cc:288] T b47b28a4766a425e8b9bdfea882f979d P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: Requested vote from peers 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:18.661512 13184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b47b28a4766a425e8b9bdfea882f979d" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770"
I20240503 13:56:18.661540 13052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b47b28a4766a425e8b9bdfea882f979d" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e"
I20240503 13:56:18.661607 13184 raft_consensus.cc:3037] T b47b28a4766a425e8b9bdfea882f979d P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:18.661653 13052 raft_consensus.cc:3037] T b47b28a4766a425e8b9bdfea882f979d P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:18.662297 13184 raft_consensus.cc:2454] T b47b28a4766a425e8b9bdfea882f979d P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 1.
I20240503 13:56:18.662381 13052 raft_consensus.cc:2454] T b47b28a4766a425e8b9bdfea882f979d P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 1.
I20240503 13:56:18.662585 13252 leader_election.cc:302] T b47b28a4766a425e8b9bdfea882f979d P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ccf5a4ac450414c9d3579bd5800b770, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:18.662793 14113 raft_consensus.cc:2781] T b47b28a4766a425e8b9bdfea882f979d P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:18.662871 14113 raft_consensus.cc:686] T b47b28a4766a425e8b9bdfea882f979d P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Becoming Leader. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Running, Role: LEADER
I20240503 13:56:18.662981 14113 consensus_queue.cc:237] T b47b28a4766a425e8b9bdfea882f979d P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:18.663900 12911 catalog_manager.cc:5484] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 reported cstate change: term changed from 0 to 1, leader changed from <none> to ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67). New cstate: current_term: 1 leader_uuid: "ba0194098aa84449ae17e8a0d4c43703" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } }
I20240503 13:56:18.667196 12911 catalog_manager.cc:5484] T b47b28a4766a425e8b9bdfea882f979d P ba0194098aa84449ae17e8a0d4c43703 reported cstate change: term changed from 0 to 1, leader changed from <none> to ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67). New cstate: current_term: 1 leader_uuid: "ba0194098aa84449ae17e8a0d4c43703" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:18.786993 14113 raft_consensus.cc:482] T 9eb0d911701b4b2bb33a6b7ab59f08aa P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:18.787158 14113 raft_consensus.cc:504] T 9eb0d911701b4b2bb33a6b7ab59f08aa P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:18.787565 14113 leader_election.cc:288] T 9eb0d911701b4b2bb33a6b7ab59f08aa P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387), 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:18.788100 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eb0d911701b4b2bb33a6b7ab59f08aa" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
I20240503 13:56:18.788095 13184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eb0d911701b4b2bb33a6b7ab59f08aa" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770" is_pre_election: true
I20240503 13:56:18.788239 13184 raft_consensus.cc:2454] T 9eb0d911701b4b2bb33a6b7ab59f08aa P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 0.
I20240503 13:56:18.788239 13446 raft_consensus.cc:2454] T 9eb0d911701b4b2bb33a6b7ab59f08aa P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 0.
I20240503 13:56:18.788630 13252 leader_election.cc:302] T 9eb0d911701b4b2bb33a6b7ab59f08aa P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ccf5a4ac450414c9d3579bd5800b770, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:18.788817 14365 raft_consensus.cc:2781] T 9eb0d911701b4b2bb33a6b7ab59f08aa P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:18.788878 14365 raft_consensus.cc:482] T 9eb0d911701b4b2bb33a6b7ab59f08aa P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:18.788915 14365 raft_consensus.cc:3037] T 9eb0d911701b4b2bb33a6b7ab59f08aa P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:18.789850 14365 raft_consensus.cc:504] T 9eb0d911701b4b2bb33a6b7ab59f08aa P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:18.790156 14365 leader_election.cc:288] T 9eb0d911701b4b2bb33a6b7ab59f08aa P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: Requested vote from peers 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387), 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:18.790501 13184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eb0d911701b4b2bb33a6b7ab59f08aa" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770"
I20240503 13:56:18.790632 13184 raft_consensus.cc:3037] T 9eb0d911701b4b2bb33a6b7ab59f08aa P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:18.790621 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eb0d911701b4b2bb33a6b7ab59f08aa" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
I20240503 13:56:18.790722 13446 raft_consensus.cc:3037] T 9eb0d911701b4b2bb33a6b7ab59f08aa P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:18.791620 13446 raft_consensus.cc:2454] T 9eb0d911701b4b2bb33a6b7ab59f08aa P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 1.
I20240503 13:56:18.791620 13184 raft_consensus.cc:2454] T 9eb0d911701b4b2bb33a6b7ab59f08aa P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 1.
I20240503 13:56:18.791900 13252 leader_election.cc:302] T 9eb0d911701b4b2bb33a6b7ab59f08aa P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ccf5a4ac450414c9d3579bd5800b770, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:18.792093 14365 raft_consensus.cc:2781] T 9eb0d911701b4b2bb33a6b7ab59f08aa P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:18.792177 14365 raft_consensus.cc:686] T 9eb0d911701b4b2bb33a6b7ab59f08aa P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Becoming Leader. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Running, Role: LEADER
I20240503 13:56:18.792511 14365 consensus_queue.cc:237] T 9eb0d911701b4b2bb33a6b7ab59f08aa P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:18.794209 12911 catalog_manager.cc:5484] T 9eb0d911701b4b2bb33a6b7ab59f08aa P ba0194098aa84449ae17e8a0d4c43703 reported cstate change: term changed from 0 to 1, leader changed from <none> to ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67). New cstate: current_term: 1 leader_uuid: "ba0194098aa84449ae17e8a0d4c43703" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
W20240503 13:56:18.935811 14406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:56:18.936070 14406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:56:18.939364 14406 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240503 13:56:19.026079 13184 raft_consensus.cc:1261] T 9eb0d911701b4b2bb33a6b7ab59f08aa P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:19.026054 13446 raft_consensus.cc:1261] T 9eb0d911701b4b2bb33a6b7ab59f08aa P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:19.026506 14365 consensus_queue.cc:1035] T 9eb0d911701b4b2bb33a6b7ab59f08aa P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:19.026749 14113 consensus_queue.cc:1035] T 9eb0d911701b4b2bb33a6b7ab59f08aa P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:19.027997 13183 raft_consensus.cc:1261] T bb4f0bb47a29437090750c4486df2f02 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:19.028363 14053 consensus_queue.cc:1035] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:19.028887 13184 raft_consensus.cc:1261] T b47b28a4766a425e8b9bdfea882f979d P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:19.030730 13052 raft_consensus.cc:1261] T b47b28a4766a425e8b9bdfea882f979d P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:19.031118 13445 raft_consensus.cc:1261] T bb4f0bb47a29437090750c4486df2f02 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:19.031687 14053 consensus_queue.cc:1035] T b47b28a4766a425e8b9bdfea882f979d P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:19.031924 14438 consensus_queue.cc:1035] T b47b28a4766a425e8b9bdfea882f979d P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:19.032181 14053 consensus_queue.cc:1035] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:19.066748 14439 mvcc.cc:205] Tried to move back new op lower bound from 7023593795691405312 to 7023593794204889088. Current Snapshot: MvccSnapshot[applied={T|T < 7023593795691405312}]
INSERT report
rows total: 2000
time total: 110.101 ms
time per row: 0.0550504 ms
I20240503 13:56:19.091818 12912 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:56:19.097229 13425 tablet_service.cc:1467] Processing CreateTablet for tablet 11f710cf381f4166b9625a6bb8646a5f (DEFAULT_TABLE table=rebalance_test_table_6 [id=89b1c278366c4030ac7c6b90c882c67e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:19.097566 13425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11f710cf381f4166b9625a6bb8646a5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:19.097702 13426 tablet_service.cc:1467] Processing CreateTablet for tablet b923fd3f226e409b813bb4e63325a7fc (DEFAULT_TABLE table=rebalance_test_table_6 [id=89b1c278366c4030ac7c6b90c882c67e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:19.097942 13426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b923fd3f226e409b813bb4e63325a7fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:19.098290 13294 tablet_service.cc:1467] Processing CreateTablet for tablet b923fd3f226e409b813bb4e63325a7fc (DEFAULT_TABLE table=rebalance_test_table_6 [id=89b1c278366c4030ac7c6b90c882c67e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:19.098558 13294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b923fd3f226e409b813bb4e63325a7fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:19.099895 14465 tablet_bootstrap.cc:492] T 11f710cf381f4166b9625a6bb8646a5f P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:19.100139 13164 tablet_service.cc:1467] Processing CreateTablet for tablet 11f710cf381f4166b9625a6bb8646a5f (DEFAULT_TABLE table=rebalance_test_table_6 [id=89b1c278366c4030ac7c6b90c882c67e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:19.100442 13164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11f710cf381f4166b9625a6bb8646a5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:19.100914 14465 tablet_bootstrap.cc:654] T 11f710cf381f4166b9625a6bb8646a5f P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:19.101016 13165 tablet_service.cc:1467] Processing CreateTablet for tablet f812304076e14c2db3dc02d1ff5766fb (DEFAULT_TABLE table=rebalance_test_table_6 [id=89b1c278366c4030ac7c6b90c882c67e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:19.101239 13165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f812304076e14c2db3dc02d1ff5766fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:19.102535 14468 tablet_bootstrap.cc:492] T 11f710cf381f4166b9625a6bb8646a5f P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:19.102901 14466 tablet_bootstrap.cc:492] T b923fd3f226e409b813bb4e63325a7fc P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:19.103449 14468 tablet_bootstrap.cc:654] T 11f710cf381f4166b9625a6bb8646a5f P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:19.103734 14466 tablet_bootstrap.cc:654] T b923fd3f226e409b813bb4e63325a7fc P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:19.103808 13034 tablet_service.cc:1467] Processing CreateTablet for tablet 11f710cf381f4166b9625a6bb8646a5f (DEFAULT_TABLE table=rebalance_test_table_6 [id=89b1c278366c4030ac7c6b90c882c67e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:19.104090 13034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11f710cf381f4166b9625a6bb8646a5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:19.104772 13032 tablet_service.cc:1467] Processing CreateTablet for tablet f812304076e14c2db3dc02d1ff5766fb (DEFAULT_TABLE table=rebalance_test_table_6 [id=89b1c278366c4030ac7c6b90c882c67e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:19.104856 14468 tablet_bootstrap.cc:492] T 11f710cf381f4166b9625a6bb8646a5f P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:19.104930 14468 ts_tablet_manager.cc:1367] T 11f710cf381f4166b9625a6bb8646a5f P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:56:19.104985 13032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f812304076e14c2db3dc02d1ff5766fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:19.105372 14468 raft_consensus.cc:348] T 11f710cf381f4166b9625a6bb8646a5f P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:19.105468 14468 raft_consensus.cc:374] T 11f710cf381f4166b9625a6bb8646a5f P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:19.105489 14468 raft_consensus.cc:729] T 11f710cf381f4166b9625a6bb8646a5f P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:19.105595 14468 consensus_queue.cc:260] T 11f710cf381f4166b9625a6bb8646a5f P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:19.105841 14468 ts_tablet_manager.cc:1398] T 11f710cf381f4166b9625a6bb8646a5f P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:19.105970 14468 tablet_bootstrap.cc:492] T f812304076e14c2db3dc02d1ff5766fb P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:19.106787 14468 tablet_bootstrap.cc:654] T f812304076e14c2db3dc02d1ff5766fb P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:19.107036 13033 tablet_service.cc:1467] Processing CreateTablet for tablet b923fd3f226e409b813bb4e63325a7fc (DEFAULT_TABLE table=rebalance_test_table_6 [id=89b1c278366c4030ac7c6b90c882c67e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:19.107231 13033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b923fd3f226e409b813bb4e63325a7fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:19.107568 14471 tablet_bootstrap.cc:492] T f812304076e14c2db3dc02d1ff5766fb P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:19.108712 14471 tablet_bootstrap.cc:654] T f812304076e14c2db3dc02d1ff5766fb P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:19.109937 14468 tablet_bootstrap.cc:492] T f812304076e14c2db3dc02d1ff5766fb P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:19.110026 14468 ts_tablet_manager.cc:1367] T f812304076e14c2db3dc02d1ff5766fb P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 13:56:19.110133 13425 tablet_service.cc:1467] Processing CreateTablet for tablet f812304076e14c2db3dc02d1ff5766fb (DEFAULT_TABLE table=rebalance_test_table_6 [id=89b1c278366c4030ac7c6b90c882c67e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:19.110389 13425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f812304076e14c2db3dc02d1ff5766fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:19.110459 14468 raft_consensus.cc:348] T f812304076e14c2db3dc02d1ff5766fb P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:19.110563 14468 raft_consensus.cc:374] T f812304076e14c2db3dc02d1ff5766fb P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:19.110601 14468 raft_consensus.cc:729] T f812304076e14c2db3dc02d1ff5766fb P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:19.110725 14468 consensus_queue.cc:260] T f812304076e14c2db3dc02d1ff5766fb P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:19.111014 14468 ts_tablet_manager.cc:1398] T f812304076e14c2db3dc02d1ff5766fb P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:19.113036 14466 tablet_bootstrap.cc:492] T b923fd3f226e409b813bb4e63325a7fc P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:19.113126 14466 ts_tablet_manager.cc:1367] T b923fd3f226e409b813bb4e63325a7fc P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20240503 13:56:19.113503 14466 raft_consensus.cc:348] T b923fd3f226e409b813bb4e63325a7fc P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:19.113598 14466 raft_consensus.cc:374] T b923fd3f226e409b813bb4e63325a7fc P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:19.113626 14466 raft_consensus.cc:729] T b923fd3f226e409b813bb4e63325a7fc P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:19.113736 14466 consensus_queue.cc:260] T b923fd3f226e409b813bb4e63325a7fc P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:19.114014 14466 ts_tablet_manager.cc:1398] T b923fd3f226e409b813bb4e63325a7fc P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:19.120868 14465 tablet_bootstrap.cc:492] T 11f710cf381f4166b9625a6bb8646a5f P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:19.120998 14465 ts_tablet_manager.cc:1367] T 11f710cf381f4166b9625a6bb8646a5f P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.002s
I20240503 13:56:19.121506 14465 raft_consensus.cc:348] T 11f710cf381f4166b9625a6bb8646a5f P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:19.121613 14465 raft_consensus.cc:374] T 11f710cf381f4166b9625a6bb8646a5f P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:19.121642 14465 raft_consensus.cc:729] T 11f710cf381f4166b9625a6bb8646a5f P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:19.121768 14465 consensus_queue.cc:260] T 11f710cf381f4166b9625a6bb8646a5f P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:19.122043 14465 ts_tablet_manager.cc:1398] T 11f710cf381f4166b9625a6bb8646a5f P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:19.122241 14465 tablet_bootstrap.cc:492] T f812304076e14c2db3dc02d1ff5766fb P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:19.123085 14465 tablet_bootstrap.cc:654] T f812304076e14c2db3dc02d1ff5766fb P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:19.124181 14465 tablet_bootstrap.cc:492] T f812304076e14c2db3dc02d1ff5766fb P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:19.124258 14465 ts_tablet_manager.cc:1367] T f812304076e14c2db3dc02d1ff5766fb P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:56:19.124949 14465 raft_consensus.cc:348] T f812304076e14c2db3dc02d1ff5766fb P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:19.125061 14465 raft_consensus.cc:374] T f812304076e14c2db3dc02d1ff5766fb P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:19.125090 14465 raft_consensus.cc:729] T f812304076e14c2db3dc02d1ff5766fb P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:19.125208 14465 consensus_queue.cc:260] T f812304076e14c2db3dc02d1ff5766fb P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:19.125468 14465 ts_tablet_manager.cc:1398] T f812304076e14c2db3dc02d1ff5766fb P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:19.125650 14465 tablet_bootstrap.cc:492] T b923fd3f226e409b813bb4e63325a7fc P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:19.126380 14465 tablet_bootstrap.cc:654] T b923fd3f226e409b813bb4e63325a7fc P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:19.127380 14465 tablet_bootstrap.cc:492] T b923fd3f226e409b813bb4e63325a7fc P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:19.127455 14465 ts_tablet_manager.cc:1367] T b923fd3f226e409b813bb4e63325a7fc P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:19.127838 14465 raft_consensus.cc:348] T b923fd3f226e409b813bb4e63325a7fc P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:19.127938 14465 raft_consensus.cc:374] T b923fd3f226e409b813bb4e63325a7fc P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:19.128005 14465 raft_consensus.cc:729] T b923fd3f226e409b813bb4e63325a7fc P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:19.128144 14465 consensus_queue.cc:260] T b923fd3f226e409b813bb4e63325a7fc P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:19.128418 14465 ts_tablet_manager.cc:1398] T b923fd3f226e409b813bb4e63325a7fc P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:19.128862 14471 tablet_bootstrap.cc:492] T f812304076e14c2db3dc02d1ff5766fb P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:19.128969 14471 ts_tablet_manager.cc:1367] T f812304076e14c2db3dc02d1ff5766fb P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.000s
I20240503 13:56:19.129408 14471 raft_consensus.cc:348] T f812304076e14c2db3dc02d1ff5766fb P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:19.129571 14471 raft_consensus.cc:374] T f812304076e14c2db3dc02d1ff5766fb P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:19.129607 14471 raft_consensus.cc:729] T f812304076e14c2db3dc02d1ff5766fb P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:19.129719 14471 consensus_queue.cc:260] T f812304076e14c2db3dc02d1ff5766fb P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:19.129969 14471 ts_tablet_manager.cc:1398] T f812304076e14c2db3dc02d1ff5766fb P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:19.130133 14471 tablet_bootstrap.cc:492] T b923fd3f226e409b813bb4e63325a7fc P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:19.130249 14410 raft_consensus.cc:482] T 11f710cf381f4166b9625a6bb8646a5f P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:19.130312 14410 raft_consensus.cc:504] T 11f710cf381f4166b9625a6bb8646a5f P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:19.130666 14410 leader_election.cc:288] T 11f710cf381f4166b9625a6bb8646a5f P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:19.130980 14471 tablet_bootstrap.cc:654] T b923fd3f226e409b813bb4e63325a7fc P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:19.131054 13052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11f710cf381f4166b9625a6bb8646a5f" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e" is_pre_election: true
I20240503 13:56:19.131346 13185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11f710cf381f4166b9625a6bb8646a5f" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770" is_pre_election: true
I20240503 13:56:19.131475 13185 raft_consensus.cc:2454] T 11f710cf381f4166b9625a6bb8646a5f P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 0.
W20240503 13:56:19.131500 13381 leader_election.cc:341] T 11f710cf381f4166b9625a6bb8646a5f P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695): Illegal state: must be running to vote when last-logged opid is not known
I20240503 13:56:19.131816 13383 leader_election.cc:302] T 11f710cf381f4166b9625a6bb8646a5f P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, 8ccf5a4ac450414c9d3579bd5800b770; no voters: 60ba3ffe3b7746469834cd91c84be14e
I20240503 13:56:19.132041 14471 tablet_bootstrap.cc:492] T b923fd3f226e409b813bb4e63325a7fc P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:19.132113 14471 ts_tablet_manager.cc:1367] T b923fd3f226e409b813bb4e63325a7fc P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:19.132189 14410 raft_consensus.cc:2781] T 11f710cf381f4166b9625a6bb8646a5f P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:19.132243 14410 raft_consensus.cc:482] T 11f710cf381f4166b9625a6bb8646a5f P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:19.132277 14410 raft_consensus.cc:3037] T 11f710cf381f4166b9625a6bb8646a5f P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:19.132449 14471 raft_consensus.cc:348] T b923fd3f226e409b813bb4e63325a7fc P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:19.132563 14471 raft_consensus.cc:374] T b923fd3f226e409b813bb4e63325a7fc P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:19.132593 14471 raft_consensus.cc:729] T b923fd3f226e409b813bb4e63325a7fc P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:19.132684 14471 consensus_queue.cc:260] T b923fd3f226e409b813bb4e63325a7fc P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:19.132865 14471 ts_tablet_manager.cc:1398] T b923fd3f226e409b813bb4e63325a7fc P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:19.132993 14471 tablet_bootstrap.cc:492] T 11f710cf381f4166b9625a6bb8646a5f P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:19.133268 14410 raft_consensus.cc:504] T 11f710cf381f4166b9625a6bb8646a5f P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:19.133591 14410 leader_election.cc:288] T 11f710cf381f4166b9625a6bb8646a5f P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 election: Requested vote from peers 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:19.133749 14471 tablet_bootstrap.cc:654] T 11f710cf381f4166b9625a6bb8646a5f P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:19.133853 13052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11f710cf381f4166b9625a6bb8646a5f" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e"
W20240503 13:56:19.134193 13381 leader_election.cc:341] T 11f710cf381f4166b9625a6bb8646a5f P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695): Illegal state: must be running to vote when last-logged opid is not known
I20240503 13:56:19.134655 13185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11f710cf381f4166b9625a6bb8646a5f" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770"
I20240503 13:56:19.134778 13185 raft_consensus.cc:3037] T 11f710cf381f4166b9625a6bb8646a5f P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:19.134889 14471 tablet_bootstrap.cc:492] T 11f710cf381f4166b9625a6bb8646a5f P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:19.134960 14471 ts_tablet_manager.cc:1367] T 11f710cf381f4166b9625a6bb8646a5f P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:56:19.135361 14471 raft_consensus.cc:348] T 11f710cf381f4166b9625a6bb8646a5f P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:19.135473 14471 raft_consensus.cc:374] T 11f710cf381f4166b9625a6bb8646a5f P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:19.135509 14471 raft_consensus.cc:729] T 11f710cf381f4166b9625a6bb8646a5f P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:19.135658 13185 raft_consensus.cc:2454] T 11f710cf381f4166b9625a6bb8646a5f P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 1.
I20240503 13:56:19.135738 14471 consensus_queue.cc:260] T 11f710cf381f4166b9625a6bb8646a5f P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:19.136011 13383 leader_election.cc:302] T 11f710cf381f4166b9625a6bb8646a5f P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, 8ccf5a4ac450414c9d3579bd5800b770; no voters: 60ba3ffe3b7746469834cd91c84be14e
I20240503 13:56:19.136067 14471 ts_tablet_manager.cc:1398] T 11f710cf381f4166b9625a6bb8646a5f P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:19.136173 14410 raft_consensus.cc:2781] T 11f710cf381f4166b9625a6bb8646a5f P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:19.136252 14410 raft_consensus.cc:686] T 11f710cf381f4166b9625a6bb8646a5f P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 LEADER]: Becoming Leader. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Running, Role: LEADER
I20240503 13:56:19.136363 14410 consensus_queue.cc:237] T 11f710cf381f4166b9625a6bb8646a5f P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:19.138136 12912 catalog_manager.cc:5484] T 11f710cf381f4166b9625a6bb8646a5f P 7690ee79577c45e3a2c8f8b552c0bff8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68). New cstate: current_term: 1 leader_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } }
I20240503 13:56:19.153021 14230 raft_consensus.cc:482] T 11f710cf381f4166b9625a6bb8646a5f P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:19.153155 14230 raft_consensus.cc:504] T 11f710cf381f4166b9625a6bb8646a5f P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:19.153605 14230 leader_election.cc:288] T 11f710cf381f4166b9625a6bb8646a5f P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:19.154070 13185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11f710cf381f4166b9625a6bb8646a5f" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770" is_pre_election: true
I20240503 13:56:19.154271 13185 raft_consensus.cc:2379] T 11f710cf381f4166b9625a6bb8646a5f P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60ba3ffe3b7746469834cd91c84be14e in current term 1: Already voted for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in this term.
I20240503 13:56:19.157706 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11f710cf381f4166b9625a6bb8646a5f" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
I20240503 13:56:19.158072 12987 leader_election.cc:302] T 11f710cf381f4166b9625a6bb8646a5f P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e; no voters: 7690ee79577c45e3a2c8f8b552c0bff8, 8ccf5a4ac450414c9d3579bd5800b770
I20240503 13:56:19.158233 14230 raft_consensus.cc:3037] T 11f710cf381f4166b9625a6bb8646a5f P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:19.159080 14230 raft_consensus.cc:2726] T 11f710cf381f4166b9625a6bb8646a5f P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 13:56:19.302752 14049 raft_consensus.cc:482] T f812304076e14c2db3dc02d1ff5766fb P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:19.302891 14049 raft_consensus.cc:504] T f812304076e14c2db3dc02d1ff5766fb P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:19.303319 14049 leader_election.cc:288] T f812304076e14c2db3dc02d1ff5766fb P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:19.303735 13052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f812304076e14c2db3dc02d1ff5766fb" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e" is_pre_election: true
I20240503 13:56:19.303735 13185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f812304076e14c2db3dc02d1ff5766fb" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770" is_pre_election: true
I20240503 13:56:19.303893 13185 raft_consensus.cc:2454] T f812304076e14c2db3dc02d1ff5766fb P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 0.
I20240503 13:56:19.303891 13052 raft_consensus.cc:2454] T f812304076e14c2db3dc02d1ff5766fb P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 0.
I20240503 13:56:19.304208 13381 leader_election.cc:302] T f812304076e14c2db3dc02d1ff5766fb P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 7690ee79577c45e3a2c8f8b552c0bff8; no voters:
I20240503 13:56:19.304386 14049 raft_consensus.cc:2781] T f812304076e14c2db3dc02d1ff5766fb P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:19.304463 14049 raft_consensus.cc:482] T f812304076e14c2db3dc02d1ff5766fb P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:19.304530 14049 raft_consensus.cc:3037] T f812304076e14c2db3dc02d1ff5766fb P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:19.305359 14049 raft_consensus.cc:504] T f812304076e14c2db3dc02d1ff5766fb P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:19.305759 14049 leader_election.cc:288] T f812304076e14c2db3dc02d1ff5766fb P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 election: Requested vote from peers 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:19.306128 13185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f812304076e14c2db3dc02d1ff5766fb" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770"
I20240503 13:56:19.306247 13185 raft_consensus.cc:3037] T f812304076e14c2db3dc02d1ff5766fb P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:19.306752 13052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f812304076e14c2db3dc02d1ff5766fb" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e"
I20240503 13:56:19.306864 13052 raft_consensus.cc:3037] T f812304076e14c2db3dc02d1ff5766fb P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:19.307253 13185 raft_consensus.cc:2454] T f812304076e14c2db3dc02d1ff5766fb P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 1.
I20240503 13:56:19.307760 13052 raft_consensus.cc:2454] T f812304076e14c2db3dc02d1ff5766fb P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 1.
I20240503 13:56:19.308068 13381 leader_election.cc:302] T f812304076e14c2db3dc02d1ff5766fb P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 7690ee79577c45e3a2c8f8b552c0bff8; no voters:
I20240503 13:56:19.308348 14410 raft_consensus.cc:2781] T f812304076e14c2db3dc02d1ff5766fb P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:19.308446 14410 raft_consensus.cc:686] T f812304076e14c2db3dc02d1ff5766fb P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 LEADER]: Becoming Leader. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Running, Role: LEADER
I20240503 13:56:19.308604 14410 consensus_queue.cc:237] T f812304076e14c2db3dc02d1ff5766fb P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:19.310286 12903 catalog_manager.cc:5484] T f812304076e14c2db3dc02d1ff5766fb P 7690ee79577c45e3a2c8f8b552c0bff8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68). New cstate: current_term: 1 leader_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } }
I20240503 13:56:19.333830 14230 raft_consensus.cc:482] T b923fd3f226e409b813bb4e63325a7fc P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:19.333983 14230 raft_consensus.cc:504] T b923fd3f226e409b813bb4e63325a7fc P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:19.334481 14230 leader_election.cc:288] T b923fd3f226e409b813bb4e63325a7fc P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279), 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:19.334931 13315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b923fd3f226e409b813bb4e63325a7fc" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703" is_pre_election: true
I20240503 13:56:19.334931 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b923fd3f226e409b813bb4e63325a7fc" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
I20240503 13:56:19.335088 13315 raft_consensus.cc:2454] T b923fd3f226e409b813bb4e63325a7fc P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 0.
I20240503 13:56:19.335101 13446 raft_consensus.cc:2454] T b923fd3f226e409b813bb4e63325a7fc P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 0.
I20240503 13:56:19.335367 12987 leader_election.cc:302] T b923fd3f226e409b813bb4e63325a7fc P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 7690ee79577c45e3a2c8f8b552c0bff8; no voters:
I20240503 13:56:19.335544 14230 raft_consensus.cc:2781] T b923fd3f226e409b813bb4e63325a7fc P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:19.335602 14230 raft_consensus.cc:482] T b923fd3f226e409b813bb4e63325a7fc P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:19.335649 14230 raft_consensus.cc:3037] T b923fd3f226e409b813bb4e63325a7fc P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:19.336555 14230 raft_consensus.cc:504] T b923fd3f226e409b813bb4e63325a7fc P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:19.336874 14230 leader_election.cc:288] T b923fd3f226e409b813bb4e63325a7fc P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 election: Requested vote from peers ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279), 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:19.337220 13315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b923fd3f226e409b813bb4e63325a7fc" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703"
I20240503 13:56:19.337293 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b923fd3f226e409b813bb4e63325a7fc" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
I20240503 13:56:19.337352 13315 raft_consensus.cc:3037] T b923fd3f226e409b813bb4e63325a7fc P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:19.337422 13446 raft_consensus.cc:3037] T b923fd3f226e409b813bb4e63325a7fc P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:19.338104 13315 raft_consensus.cc:2454] T b923fd3f226e409b813bb4e63325a7fc P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 1.
I20240503 13:56:19.338109 13446 raft_consensus.cc:2454] T b923fd3f226e409b813bb4e63325a7fc P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 1.
I20240503 13:56:19.338388 12987 leader_election.cc:302] T b923fd3f226e409b813bb4e63325a7fc P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 7690ee79577c45e3a2c8f8b552c0bff8; no voters:
I20240503 13:56:19.338552 14230 raft_consensus.cc:2781] T b923fd3f226e409b813bb4e63325a7fc P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:19.338788 14230 raft_consensus.cc:686] T b923fd3f226e409b813bb4e63325a7fc P 60ba3ffe3b7746469834cd91c84be14e [term 1 LEADER]: Becoming Leader. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Running, Role: LEADER
I20240503 13:56:19.338932 14230 consensus_queue.cc:237] T b923fd3f226e409b813bb4e63325a7fc P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:19.340723 12903 catalog_manager.cc:5484] T b923fd3f226e409b813bb4e63325a7fc P 60ba3ffe3b7746469834cd91c84be14e reported cstate change: term changed from 0 to 1, leader changed from <none> to 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65). New cstate: current_term: 1 leader_uuid: "60ba3ffe3b7746469834cd91c84be14e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } }
W20240503 13:56:19.455741 14485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:56:19.456051 14485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:56:19.459266 14485 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240503 13:56:19.548161 13446 raft_consensus.cc:1261] T b923fd3f226e409b813bb4e63325a7fc P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer 60ba3ffe3b7746469834cd91c84be14e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:19.548705 14483 consensus_queue.cc:1035] T b923fd3f226e409b813bb4e63325a7fc P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:19.549005 13185 raft_consensus.cc:1261] T 11f710cf381f4166b9625a6bb8646a5f P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Refusing update from remote peer 7690ee79577c45e3a2c8f8b552c0bff8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:19.549376 13052 raft_consensus.cc:1261] T 11f710cf381f4166b9625a6bb8646a5f P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Refusing update from remote peer 7690ee79577c45e3a2c8f8b552c0bff8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:19.549420 13315 raft_consensus.cc:1261] T b923fd3f226e409b813bb4e63325a7fc P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Refusing update from remote peer 60ba3ffe3b7746469834cd91c84be14e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:19.549801 14230 consensus_queue.cc:1035] T b923fd3f226e409b813bb4e63325a7fc P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:19.549904 14481 consensus_queue.cc:1035] T 11f710cf381f4166b9625a6bb8646a5f P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:19.551635 14049 consensus_queue.cc:1035] T 11f710cf381f4166b9625a6bb8646a5f P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:19.553514 13184 raft_consensus.cc:1261] T f812304076e14c2db3dc02d1ff5766fb P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Refusing update from remote peer 7690ee79577c45e3a2c8f8b552c0bff8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:19.553812 13053 raft_consensus.cc:1261] T f812304076e14c2db3dc02d1ff5766fb P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Refusing update from remote peer 7690ee79577c45e3a2c8f8b552c0bff8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:19.553902 14481 consensus_queue.cc:1035] T f812304076e14c2db3dc02d1ff5766fb P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:19.558988 14517 consensus_queue.cc:1035] T f812304076e14c2db3dc02d1ff5766fb P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:19.578200 14445 mvcc.cc:205] Tried to move back new op lower bound from 7023593797825540096 to 7023593796973592576. Current Snapshot: MvccSnapshot[applied={T|T < 7023593797825540096}]
I20240503 13:56:19.596041 14451 mvcc.cc:205] Tried to move back new op lower bound from 7023593797838983168 to 7023593796849258496. Current Snapshot: MvccSnapshot[applied={T|T < 7023593797838983168 or (T in {7023593797838983168})}]
INSERT report
rows total: 2000
time total: 114.035 ms
time per row: 0.0570177 ms
I20240503 13:56:19.616547 12906 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:56:19.631318 13034 tablet_service.cc:1467] Processing CreateTablet for tablet 30d2b5ede30744cfb46ec1be232a1405 (DEFAULT_TABLE table=rebalance_test_table_7 [id=5e6f2f3703e1424ab61d8fe49e6266dd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:19.631433 13165 tablet_service.cc:1467] Processing CreateTablet for tablet 30d2b5ede30744cfb46ec1be232a1405 (DEFAULT_TABLE table=rebalance_test_table_7 [id=5e6f2f3703e1424ab61d8fe49e6266dd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:19.631676 13165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30d2b5ede30744cfb46ec1be232a1405. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:19.631682 13034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30d2b5ede30744cfb46ec1be232a1405. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:19.631891 13294 tablet_service.cc:1467] Processing CreateTablet for tablet 30d2b5ede30744cfb46ec1be232a1405 (DEFAULT_TABLE table=rebalance_test_table_7 [id=5e6f2f3703e1424ab61d8fe49e6266dd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:19.632050 13425 tablet_service.cc:1467] Processing CreateTablet for tablet 258d1ec1f0694c15a4d41bd064f35b4e (DEFAULT_TABLE table=rebalance_test_table_7 [id=5e6f2f3703e1424ab61d8fe49e6266dd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:19.632182 13294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30d2b5ede30744cfb46ec1be232a1405. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:19.632279 13425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 258d1ec1f0694c15a4d41bd064f35b4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:19.633961 14471 tablet_bootstrap.cc:492] T 30d2b5ede30744cfb46ec1be232a1405 P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:19.634150 13426 tablet_service.cc:1467] Processing CreateTablet for tablet 7b57f64985b74e5fa75a5603ab8cc4bf (DEFAULT_TABLE table=rebalance_test_table_7 [id=5e6f2f3703e1424ab61d8fe49e6266dd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:19.634367 13426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b57f64985b74e5fa75a5603ab8cc4bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:19.634760 14525 tablet_bootstrap.cc:492] T 258d1ec1f0694c15a4d41bd064f35b4e P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:19.634940 14471 tablet_bootstrap.cc:654] T 30d2b5ede30744cfb46ec1be232a1405 P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:19.635427 14525 tablet_bootstrap.cc:654] T 258d1ec1f0694c15a4d41bd064f35b4e P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:19.635684 14526 tablet_bootstrap.cc:492] T 30d2b5ede30744cfb46ec1be232a1405 P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:19.636334 14471 tablet_bootstrap.cc:492] T 30d2b5ede30744cfb46ec1be232a1405 P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:19.636430 14526 tablet_bootstrap.cc:654] T 30d2b5ede30744cfb46ec1be232a1405 P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:19.636422 14471 ts_tablet_manager.cc:1367] T 30d2b5ede30744cfb46ec1be232a1405 P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:56:19.636893 13165 tablet_service.cc:1467] Processing CreateTablet for tablet 258d1ec1f0694c15a4d41bd064f35b4e (DEFAULT_TABLE table=rebalance_test_table_7 [id=5e6f2f3703e1424ab61d8fe49e6266dd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:19.636906 13294 tablet_service.cc:1467] Processing CreateTablet for tablet 7b57f64985b74e5fa75a5603ab8cc4bf (DEFAULT_TABLE table=rebalance_test_table_7 [id=5e6f2f3703e1424ab61d8fe49e6266dd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:19.636898 14471 raft_consensus.cc:348] T 30d2b5ede30744cfb46ec1be232a1405 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:19.637004 14471 raft_consensus.cc:374] T 30d2b5ede30744cfb46ec1be232a1405 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:19.637037 14471 raft_consensus.cc:729] T 30d2b5ede30744cfb46ec1be232a1405 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:19.637136 13165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 258d1ec1f0694c15a4d41bd064f35b4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:19.637136 13294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b57f64985b74e5fa75a5603ab8cc4bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:19.637151 14471 consensus_queue.cc:260] T 30d2b5ede30744cfb46ec1be232a1405 P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:19.637401 14524 tablet_bootstrap.cc:492] T 30d2b5ede30744cfb46ec1be232a1405 P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:19.637423 14471 ts_tablet_manager.cc:1398] T 30d2b5ede30744cfb46ec1be232a1405 P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:19.637784 13034 tablet_service.cc:1467] Processing CreateTablet for tablet 7b57f64985b74e5fa75a5603ab8cc4bf (DEFAULT_TABLE table=rebalance_test_table_7 [id=5e6f2f3703e1424ab61d8fe49e6266dd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:19.638021 13034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b57f64985b74e5fa75a5603ab8cc4bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:19.638360 14524 tablet_bootstrap.cc:654] T 30d2b5ede30744cfb46ec1be232a1405 P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:19.639133 13295 tablet_service.cc:1467] Processing CreateTablet for tablet 258d1ec1f0694c15a4d41bd064f35b4e (DEFAULT_TABLE table=rebalance_test_table_7 [id=5e6f2f3703e1424ab61d8fe49e6266dd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:19.639361 13295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 258d1ec1f0694c15a4d41bd064f35b4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:19.639364 14524 tablet_bootstrap.cc:492] T 30d2b5ede30744cfb46ec1be232a1405 P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:19.639439 14524 ts_tablet_manager.cc:1367] T 30d2b5ede30744cfb46ec1be232a1405 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:19.639847 14524 raft_consensus.cc:348] T 30d2b5ede30744cfb46ec1be232a1405 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:19.639956 14524 raft_consensus.cc:374] T 30d2b5ede30744cfb46ec1be232a1405 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:19.639987 14524 raft_consensus.cc:729] T 30d2b5ede30744cfb46ec1be232a1405 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:19.640156 14471 tablet_bootstrap.cc:492] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:19.640126 14524 consensus_queue.cc:260] T 30d2b5ede30744cfb46ec1be232a1405 P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:19.640398 14524 ts_tablet_manager.cc:1398] T 30d2b5ede30744cfb46ec1be232a1405 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:19.640580 14524 tablet_bootstrap.cc:492] T 258d1ec1f0694c15a4d41bd064f35b4e P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:19.641006 14471 tablet_bootstrap.cc:654] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:19.641389 14524 tablet_bootstrap.cc:654] T 258d1ec1f0694c15a4d41bd064f35b4e P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:19.642369 14525 tablet_bootstrap.cc:492] T 258d1ec1f0694c15a4d41bd064f35b4e P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:19.642491 14525 ts_tablet_manager.cc:1367] T 258d1ec1f0694c15a4d41bd064f35b4e P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 13:56:19.643105 14524 tablet_bootstrap.cc:492] T 258d1ec1f0694c15a4d41bd064f35b4e P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:19.643119 14526 tablet_bootstrap.cc:492] T 30d2b5ede30744cfb46ec1be232a1405 P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:19.643191 14524 ts_tablet_manager.cc:1367] T 258d1ec1f0694c15a4d41bd064f35b4e P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:56:19.643194 14526 ts_tablet_manager.cc:1367] T 30d2b5ede30744cfb46ec1be232a1405 P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240503 13:56:19.642984 14525 raft_consensus.cc:348] T 258d1ec1f0694c15a4d41bd064f35b4e P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:19.643594 14524 raft_consensus.cc:348] T 258d1ec1f0694c15a4d41bd064f35b4e P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:19.643594 14526 raft_consensus.cc:348] T 30d2b5ede30744cfb46ec1be232a1405 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:19.643657 14525 raft_consensus.cc:374] T 258d1ec1f0694c15a4d41bd064f35b4e P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:19.643687 14526 raft_consensus.cc:374] T 30d2b5ede30744cfb46ec1be232a1405 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:19.643687 14524 raft_consensus.cc:374] T 258d1ec1f0694c15a4d41bd064f35b4e P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:19.643703 14525 raft_consensus.cc:729] T 258d1ec1f0694c15a4d41bd064f35b4e P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:19.643726 14526 raft_consensus.cc:729] T 30d2b5ede30744cfb46ec1be232a1405 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:19.643726 14524 raft_consensus.cc:729] T 258d1ec1f0694c15a4d41bd064f35b4e P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:19.643839 14524 consensus_queue.cc:260] T 258d1ec1f0694c15a4d41bd064f35b4e P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:19.643863 14526 consensus_queue.cc:260] T 30d2b5ede30744cfb46ec1be232a1405 P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:19.644129 14524 ts_tablet_manager.cc:1398] T 258d1ec1f0694c15a4d41bd064f35b4e P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:19.644177 14526 ts_tablet_manager.cc:1398] T 30d2b5ede30744cfb46ec1be232a1405 P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:19.644311 14526 tablet_bootstrap.cc:492] T 7b57f64985b74e5fa75a5603ab8cc4bf P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:19.643889 14525 consensus_queue.cc:260] T 258d1ec1f0694c15a4d41bd064f35b4e P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:19.644698 14525 ts_tablet_manager.cc:1398] T 258d1ec1f0694c15a4d41bd064f35b4e P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:19.644868 14525 tablet_bootstrap.cc:492] T 7b57f64985b74e5fa75a5603ab8cc4bf P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:19.645108 14526 tablet_bootstrap.cc:654] T 7b57f64985b74e5fa75a5603ab8cc4bf P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:19.645828 14525 tablet_bootstrap.cc:654] T 7b57f64985b74e5fa75a5603ab8cc4bf P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:19.646891 14471 tablet_bootstrap.cc:492] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:19.646991 14471 ts_tablet_manager.cc:1367] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240503 13:56:19.647403 14526 tablet_bootstrap.cc:492] T 7b57f64985b74e5fa75a5603ab8cc4bf P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:19.647359 14471 raft_consensus.cc:348] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:19.647449 14471 raft_consensus.cc:374] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:19.647477 14526 ts_tablet_manager.cc:1367] T 7b57f64985b74e5fa75a5603ab8cc4bf P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:56:19.647475 14471 raft_consensus.cc:729] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:19.647609 14471 consensus_queue.cc:260] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:19.647864 14471 ts_tablet_manager.cc:1398] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:19.647843 14526 raft_consensus.cc:348] T 7b57f64985b74e5fa75a5603ab8cc4bf P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:19.647930 14526 raft_consensus.cc:374] T 7b57f64985b74e5fa75a5603ab8cc4bf P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:19.647951 14526 raft_consensus.cc:729] T 7b57f64985b74e5fa75a5603ab8cc4bf P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:19.648049 14526 consensus_queue.cc:260] T 7b57f64985b74e5fa75a5603ab8cc4bf P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:19.648312 14526 ts_tablet_manager.cc:1398] T 7b57f64985b74e5fa75a5603ab8cc4bf P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:19.648646 14526 tablet_bootstrap.cc:492] T 258d1ec1f0694c15a4d41bd064f35b4e P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:19.649268 14525 tablet_bootstrap.cc:492] T 7b57f64985b74e5fa75a5603ab8cc4bf P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:19.649348 14525 ts_tablet_manager.cc:1367] T 7b57f64985b74e5fa75a5603ab8cc4bf P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 13:56:19.649485 14526 tablet_bootstrap.cc:654] T 258d1ec1f0694c15a4d41bd064f35b4e P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:19.649781 14525 raft_consensus.cc:348] T 7b57f64985b74e5fa75a5603ab8cc4bf P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:19.649909 14525 raft_consensus.cc:374] T 7b57f64985b74e5fa75a5603ab8cc4bf P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:19.649945 14525 raft_consensus.cc:729] T 7b57f64985b74e5fa75a5603ab8cc4bf P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:19.650067 14525 consensus_queue.cc:260] T 7b57f64985b74e5fa75a5603ab8cc4bf P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:19.650408 14525 ts_tablet_manager.cc:1398] T 7b57f64985b74e5fa75a5603ab8cc4bf P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:19.651124 14526 tablet_bootstrap.cc:492] T 258d1ec1f0694c15a4d41bd064f35b4e P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:19.651216 14526 ts_tablet_manager.cc:1367] T 258d1ec1f0694c15a4d41bd064f35b4e P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:56:19.651623 14526 raft_consensus.cc:348] T 258d1ec1f0694c15a4d41bd064f35b4e P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:19.651721 14526 raft_consensus.cc:374] T 258d1ec1f0694c15a4d41bd064f35b4e P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:19.651747 14526 raft_consensus.cc:729] T 258d1ec1f0694c15a4d41bd064f35b4e P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:19.651846 14526 consensus_queue.cc:260] T 258d1ec1f0694c15a4d41bd064f35b4e P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:19.652212 14526 ts_tablet_manager.cc:1398] T 258d1ec1f0694c15a4d41bd064f35b4e P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:19.692605 14437 raft_consensus.cc:482] T 258d1ec1f0694c15a4d41bd064f35b4e P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:19.692734 14437 raft_consensus.cc:504] T 258d1ec1f0694c15a4d41bd064f35b4e P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:19.693130 14437 leader_election.cc:288] T 258d1ec1f0694c15a4d41bd064f35b4e P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387), 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:19.693485 13184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "258d1ec1f0694c15a4d41bd064f35b4e" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770" is_pre_election: true
I20240503 13:56:19.693524 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "258d1ec1f0694c15a4d41bd064f35b4e" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
I20240503 13:56:19.693616 13184 raft_consensus.cc:2454] T 258d1ec1f0694c15a4d41bd064f35b4e P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 0.
I20240503 13:56:19.693665 13446 raft_consensus.cc:2454] T 258d1ec1f0694c15a4d41bd064f35b4e P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 0.
I20240503 13:56:19.693886 13252 leader_election.cc:302] T 258d1ec1f0694c15a4d41bd064f35b4e P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ccf5a4ac450414c9d3579bd5800b770, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:19.694069 14437 raft_consensus.cc:2781] T 258d1ec1f0694c15a4d41bd064f35b4e P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:19.694129 14437 raft_consensus.cc:482] T 258d1ec1f0694c15a4d41bd064f35b4e P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:19.694168 14437 raft_consensus.cc:3037] T 258d1ec1f0694c15a4d41bd064f35b4e P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:19.695005 14437 raft_consensus.cc:504] T 258d1ec1f0694c15a4d41bd064f35b4e P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:19.695326 14437 leader_election.cc:288] T 258d1ec1f0694c15a4d41bd064f35b4e P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: Requested vote from peers 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387), 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:19.695698 13184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "258d1ec1f0694c15a4d41bd064f35b4e" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770"
I20240503 13:56:19.695708 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "258d1ec1f0694c15a4d41bd064f35b4e" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
I20240503 13:56:19.695807 13184 raft_consensus.cc:3037] T 258d1ec1f0694c15a4d41bd064f35b4e P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:19.695824 13446 raft_consensus.cc:3037] T 258d1ec1f0694c15a4d41bd064f35b4e P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:19.696772 13184 raft_consensus.cc:2454] T 258d1ec1f0694c15a4d41bd064f35b4e P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 1.
I20240503 13:56:19.696785 13446 raft_consensus.cc:2454] T 258d1ec1f0694c15a4d41bd064f35b4e P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 1.
I20240503 13:56:19.697054 13249 leader_election.cc:302] T 258d1ec1f0694c15a4d41bd064f35b4e P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:19.697213 14437 raft_consensus.cc:2781] T 258d1ec1f0694c15a4d41bd064f35b4e P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:19.697299 14437 raft_consensus.cc:686] T 258d1ec1f0694c15a4d41bd064f35b4e P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Becoming Leader. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Running, Role: LEADER
I20240503 13:56:19.697430 14437 consensus_queue.cc:237] T 258d1ec1f0694c15a4d41bd064f35b4e P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:19.699118 12904 catalog_manager.cc:5484] T 258d1ec1f0694c15a4d41bd064f35b4e P ba0194098aa84449ae17e8a0d4c43703 reported cstate change: term changed from 0 to 1, leader changed from <none> to ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67). New cstate: current_term: 1 leader_uuid: "ba0194098aa84449ae17e8a0d4c43703" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } }
I20240503 13:56:19.744855 14141 raft_consensus.cc:482] T 30d2b5ede30744cfb46ec1be232a1405 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:19.745000 14141 raft_consensus.cc:504] T 30d2b5ede30744cfb46ec1be232a1405 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:19.745388 14141 leader_election.cc:288] T 30d2b5ede30744cfb46ec1be232a1405 P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279), 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:19.745932 13053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30d2b5ede30744cfb46ec1be232a1405" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e" is_pre_election: true
I20240503 13:56:19.745970 13315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30d2b5ede30744cfb46ec1be232a1405" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703" is_pre_election: true
I20240503 13:56:19.746081 13053 raft_consensus.cc:2454] T 30d2b5ede30744cfb46ec1be232a1405 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 0.
I20240503 13:56:19.746091 13315 raft_consensus.cc:2454] T 30d2b5ede30744cfb46ec1be232a1405 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 0.
I20240503 13:56:19.746377 13119 leader_election.cc:302] T 30d2b5ede30744cfb46ec1be232a1405 P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 8ccf5a4ac450414c9d3579bd5800b770; no voters:
I20240503 13:56:19.746546 14141 raft_consensus.cc:2781] T 30d2b5ede30744cfb46ec1be232a1405 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:19.746604 14141 raft_consensus.cc:482] T 30d2b5ede30744cfb46ec1be232a1405 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:19.746630 14141 raft_consensus.cc:3037] T 30d2b5ede30744cfb46ec1be232a1405 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:19.747462 14141 raft_consensus.cc:504] T 30d2b5ede30744cfb46ec1be232a1405 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:19.747790 14141 leader_election.cc:288] T 30d2b5ede30744cfb46ec1be232a1405 P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 election: Requested vote from peers ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279), 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:19.748132 13315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30d2b5ede30744cfb46ec1be232a1405" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703"
I20240503 13:56:19.748169 13053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30d2b5ede30744cfb46ec1be232a1405" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e"
I20240503 13:56:19.748256 13315 raft_consensus.cc:3037] T 30d2b5ede30744cfb46ec1be232a1405 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:19.748291 13053 raft_consensus.cc:3037] T 30d2b5ede30744cfb46ec1be232a1405 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:19.749064 13053 raft_consensus.cc:2454] T 30d2b5ede30744cfb46ec1be232a1405 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 1.
I20240503 13:56:19.749083 13315 raft_consensus.cc:2454] T 30d2b5ede30744cfb46ec1be232a1405 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 1.
I20240503 13:56:19.749394 13119 leader_election.cc:302] T 30d2b5ede30744cfb46ec1be232a1405 P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 8ccf5a4ac450414c9d3579bd5800b770; no voters:
I20240503 13:56:19.749562 14141 raft_consensus.cc:2781] T 30d2b5ede30744cfb46ec1be232a1405 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:19.749655 14141 raft_consensus.cc:686] T 30d2b5ede30744cfb46ec1be232a1405 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 LEADER]: Becoming Leader. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Running, Role: LEADER
I20240503 13:56:19.749796 14141 consensus_queue.cc:237] T 30d2b5ede30744cfb46ec1be232a1405 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:19.751607 12904 catalog_manager.cc:5484] T 30d2b5ede30744cfb46ec1be232a1405 P 8ccf5a4ac450414c9d3579bd5800b770 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66). New cstate: current_term: 1 leader_uuid: "8ccf5a4ac450414c9d3579bd5800b770" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } }
I20240503 13:56:19.833688 14230 raft_consensus.cc:482] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:19.833824 14230 raft_consensus.cc:504] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:19.834208 14230 leader_election.cc:288] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279), 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:19.834589 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b57f64985b74e5fa75a5603ab8cc4bf" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
I20240503 13:56:19.834589 13315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b57f64985b74e5fa75a5603ab8cc4bf" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703" is_pre_election: true
I20240503 13:56:19.834724 13446 raft_consensus.cc:2454] T 7b57f64985b74e5fa75a5603ab8cc4bf P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 0.
I20240503 13:56:19.834724 13315 raft_consensus.cc:2454] T 7b57f64985b74e5fa75a5603ab8cc4bf P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 0.
I20240503 13:56:19.835019 12990 leader_election.cc:302] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:19.835201 14230 raft_consensus.cc:2781] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:19.835258 14230 raft_consensus.cc:482] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:19.835286 14230 raft_consensus.cc:3037] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:19.836064 14230 raft_consensus.cc:504] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:19.836401 14230 leader_election.cc:288] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 election: Requested vote from peers ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279), 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:19.836752 13315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b57f64985b74e5fa75a5603ab8cc4bf" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703"
I20240503 13:56:19.836752 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b57f64985b74e5fa75a5603ab8cc4bf" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
I20240503 13:56:19.836876 13315 raft_consensus.cc:3037] T 7b57f64985b74e5fa75a5603ab8cc4bf P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:19.836876 13446 raft_consensus.cc:3037] T 7b57f64985b74e5fa75a5603ab8cc4bf P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:19.837776 13315 raft_consensus.cc:2454] T 7b57f64985b74e5fa75a5603ab8cc4bf P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 1.
I20240503 13:56:19.837776 13446 raft_consensus.cc:2454] T 7b57f64985b74e5fa75a5603ab8cc4bf P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 1.
I20240503 13:56:19.838083 12987 leader_election.cc:302] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 7690ee79577c45e3a2c8f8b552c0bff8; no voters:
I20240503 13:56:19.838244 14230 raft_consensus.cc:2781] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:19.838325 14230 raft_consensus.cc:686] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e [term 1 LEADER]: Becoming Leader. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Running, Role: LEADER
I20240503 13:56:19.838452 14230 consensus_queue.cc:237] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:19.840147 12904 catalog_manager.cc:5484] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e reported cstate change: term changed from 0 to 1, leader changed from <none> to 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65). New cstate: current_term: 1 leader_uuid: "60ba3ffe3b7746469834cd91c84be14e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } }
W20240503 13:56:19.982926 14539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:56:19.983178 14539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:56:19.986275 14539 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240503 13:56:20.072960 13446 raft_consensus.cc:1261] T 7b57f64985b74e5fa75a5603ab8cc4bf P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer 60ba3ffe3b7746469834cd91c84be14e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:20.073375 14230 consensus_queue.cc:1035] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:20.075165 13053 raft_consensus.cc:1261] T 30d2b5ede30744cfb46ec1be232a1405 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Refusing update from remote peer 8ccf5a4ac450414c9d3579bd5800b770: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:20.076575 13315 raft_consensus.cc:1261] T 30d2b5ede30744cfb46ec1be232a1405 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Refusing update from remote peer 8ccf5a4ac450414c9d3579bd5800b770: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:20.077903 13184 raft_consensus.cc:1261] T 258d1ec1f0694c15a4d41bd064f35b4e P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:20.078320 14438 consensus_queue.cc:1035] T 258d1ec1f0694c15a4d41bd064f35b4e P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:20.078898 13445 raft_consensus.cc:1261] T 258d1ec1f0694c15a4d41bd064f35b4e P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:20.079285 14253 consensus_queue.cc:1035] T 30d2b5ede30744cfb46ec1be232a1405 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:20.079636 14253 consensus_queue.cc:1035] T 30d2b5ede30744cfb46ec1be232a1405 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:20.079807 14438 consensus_queue.cc:1035] T 258d1ec1f0694c15a4d41bd064f35b4e P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:20.080744 13315 raft_consensus.cc:1261] T 7b57f64985b74e5fa75a5603ab8cc4bf P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Refusing update from remote peer 60ba3ffe3b7746469834cd91c84be14e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:20.085806 14483 consensus_queue.cc:1035] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:20.136108 14579 mvcc.cc:205] Tried to move back new op lower bound from 7023593799981133824 to 7023593798656512000. Current Snapshot: MvccSnapshot[applied={T|T < 7023593799981133824}]
INSERT report
rows total: 2000
time total: 120.612 ms
time per row: 0.0603058 ms
I20240503 13:56:20.151702 12911 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:56:20.157516 13165 tablet_service.cc:1467] Processing CreateTablet for tablet 973a44e737844ffb99dd3e168fcdc1d5 (DEFAULT_TABLE table=rebalance_test_table_8 [id=575bc3a99f6c434980a3decff326d982]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:20.157565 13295 tablet_service.cc:1467] Processing CreateTablet for tablet 973a44e737844ffb99dd3e168fcdc1d5 (DEFAULT_TABLE table=rebalance_test_table_8 [id=575bc3a99f6c434980a3decff326d982]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:20.157856 13165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 973a44e737844ffb99dd3e168fcdc1d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:20.157860 13295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 973a44e737844ffb99dd3e168fcdc1d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:20.157997 13034 tablet_service.cc:1467] Processing CreateTablet for tablet 973a44e737844ffb99dd3e168fcdc1d5 (DEFAULT_TABLE table=rebalance_test_table_8 [id=575bc3a99f6c434980a3decff326d982]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:20.158239 13034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 973a44e737844ffb99dd3e168fcdc1d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:20.159624 14585 tablet_bootstrap.cc:492] T 973a44e737844ffb99dd3e168fcdc1d5 P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:20.160260 13033 tablet_service.cc:1467] Processing CreateTablet for tablet 08e3096cdac34940a3ab0529cb76c3c7 (DEFAULT_TABLE table=rebalance_test_table_8 [id=575bc3a99f6c434980a3decff326d982]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:20.160274 13295 tablet_service.cc:1467] Processing CreateTablet for tablet e8e52f0e9d3340b2bd468e2c7d8bc921 (DEFAULT_TABLE table=rebalance_test_table_8 [id=575bc3a99f6c434980a3decff326d982]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:20.160478 13033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08e3096cdac34940a3ab0529cb76c3c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:20.160558 13295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8e52f0e9d3340b2bd468e2c7d8bc921. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:20.160619 14585 tablet_bootstrap.cc:654] T 973a44e737844ffb99dd3e168fcdc1d5 P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:20.161326 13426 tablet_service.cc:1467] Processing CreateTablet for tablet 08e3096cdac34940a3ab0529cb76c3c7 (DEFAULT_TABLE table=rebalance_test_table_8 [id=575bc3a99f6c434980a3decff326d982]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:20.161326 13425 tablet_service.cc:1467] Processing CreateTablet for tablet e8e52f0e9d3340b2bd468e2c7d8bc921 (DEFAULT_TABLE table=rebalance_test_table_8 [id=575bc3a99f6c434980a3decff326d982]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:20.161617 13426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08e3096cdac34940a3ab0529cb76c3c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:20.161775 13425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8e52f0e9d3340b2bd468e2c7d8bc921. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:20.162016 14585 tablet_bootstrap.cc:492] T 973a44e737844ffb99dd3e168fcdc1d5 P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:20.162098 14585 ts_tablet_manager.cc:1367] T 973a44e737844ffb99dd3e168fcdc1d5 P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 13:56:20.162465 13032 tablet_service.cc:1467] Processing CreateTablet for tablet e8e52f0e9d3340b2bd468e2c7d8bc921 (DEFAULT_TABLE table=rebalance_test_table_8 [id=575bc3a99f6c434980a3decff326d982]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:20.162528 14585 raft_consensus.cc:348] T 973a44e737844ffb99dd3e168fcdc1d5 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:20.162633 14585 raft_consensus.cc:374] T 973a44e737844ffb99dd3e168fcdc1d5 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:20.162663 14585 raft_consensus.cc:729] T 973a44e737844ffb99dd3e168fcdc1d5 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:20.162748 13032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8e52f0e9d3340b2bd468e2c7d8bc921. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:20.162786 14585 consensus_queue.cc:260] T 973a44e737844ffb99dd3e168fcdc1d5 P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:20.163079 14585 ts_tablet_manager.cc:1398] T 973a44e737844ffb99dd3e168fcdc1d5 P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:20.164373 14587 tablet_bootstrap.cc:492] T 973a44e737844ffb99dd3e168fcdc1d5 P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:20.165274 14585 tablet_bootstrap.cc:492] T e8e52f0e9d3340b2bd468e2c7d8bc921 P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:20.165753 13165 tablet_service.cc:1467] Processing CreateTablet for tablet 08e3096cdac34940a3ab0529cb76c3c7 (DEFAULT_TABLE table=rebalance_test_table_8 [id=575bc3a99f6c434980a3decff326d982]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:20.165956 14587 tablet_bootstrap.cc:654] T 973a44e737844ffb99dd3e168fcdc1d5 P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:20.166047 13165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08e3096cdac34940a3ab0529cb76c3c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:20.166129 14585 tablet_bootstrap.cc:654] T e8e52f0e9d3340b2bd468e2c7d8bc921 P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:20.166716 14589 tablet_bootstrap.cc:492] T 08e3096cdac34940a3ab0529cb76c3c7 P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:20.167191 14585 tablet_bootstrap.cc:492] T e8e52f0e9d3340b2bd468e2c7d8bc921 P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:20.167686 14589 tablet_bootstrap.cc:654] T 08e3096cdac34940a3ab0529cb76c3c7 P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:20.168051 14587 tablet_bootstrap.cc:492] T 973a44e737844ffb99dd3e168fcdc1d5 P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:20.168145 14587 ts_tablet_manager.cc:1367] T 973a44e737844ffb99dd3e168fcdc1d5 P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:56:20.167270 14585 ts_tablet_manager.cc:1367] T e8e52f0e9d3340b2bd468e2c7d8bc921 P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 13:56:20.168768 14587 raft_consensus.cc:348] T 973a44e737844ffb99dd3e168fcdc1d5 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:20.168879 14587 raft_consensus.cc:374] T 973a44e737844ffb99dd3e168fcdc1d5 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:20.168913 14587 raft_consensus.cc:729] T 973a44e737844ffb99dd3e168fcdc1d5 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:20.168910 14585 raft_consensus.cc:348] T e8e52f0e9d3340b2bd468e2c7d8bc921 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:20.168993 14585 raft_consensus.cc:374] T e8e52f0e9d3340b2bd468e2c7d8bc921 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:20.169021 14585 raft_consensus.cc:729] T e8e52f0e9d3340b2bd468e2c7d8bc921 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:20.169088 14587 consensus_queue.cc:260] T 973a44e737844ffb99dd3e168fcdc1d5 P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:20.169139 14585 consensus_queue.cc:260] T e8e52f0e9d3340b2bd468e2c7d8bc921 P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:20.169308 14586 tablet_bootstrap.cc:492] T 973a44e737844ffb99dd3e168fcdc1d5 P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:20.169700 14585 ts_tablet_manager.cc:1398] T e8e52f0e9d3340b2bd468e2c7d8bc921 P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:20.170024 14587 ts_tablet_manager.cc:1398] T 973a44e737844ffb99dd3e168fcdc1d5 P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:56:20.170167 14586 tablet_bootstrap.cc:654] T 973a44e737844ffb99dd3e168fcdc1d5 P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:20.170192 14587 tablet_bootstrap.cc:492] T 08e3096cdac34940a3ab0529cb76c3c7 P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:20.171114 14587 tablet_bootstrap.cc:654] T 08e3096cdac34940a3ab0529cb76c3c7 P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:20.172605 14589 tablet_bootstrap.cc:492] T 08e3096cdac34940a3ab0529cb76c3c7 P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:20.172701 14589 ts_tablet_manager.cc:1367] T 08e3096cdac34940a3ab0529cb76c3c7 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 13:56:20.173138 14589 raft_consensus.cc:348] T 08e3096cdac34940a3ab0529cb76c3c7 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:20.173243 14589 raft_consensus.cc:374] T 08e3096cdac34940a3ab0529cb76c3c7 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:20.173272 14589 raft_consensus.cc:729] T 08e3096cdac34940a3ab0529cb76c3c7 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:20.173406 14589 consensus_queue.cc:260] T 08e3096cdac34940a3ab0529cb76c3c7 P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:20.173740 14589 ts_tablet_manager.cc:1398] T 08e3096cdac34940a3ab0529cb76c3c7 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:20.174018 14589 tablet_bootstrap.cc:492] T e8e52f0e9d3340b2bd468e2c7d8bc921 P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:20.174139 14586 tablet_bootstrap.cc:492] T 973a44e737844ffb99dd3e168fcdc1d5 P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:20.174219 14586 ts_tablet_manager.cc:1367] T 973a44e737844ffb99dd3e168fcdc1d5 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 13:56:20.174299 14587 tablet_bootstrap.cc:492] T 08e3096cdac34940a3ab0529cb76c3c7 P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:20.174386 14587 ts_tablet_manager.cc:1367] T 08e3096cdac34940a3ab0529cb76c3c7 P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:56:20.174858 14587 raft_consensus.cc:348] T 08e3096cdac34940a3ab0529cb76c3c7 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:20.174995 14587 raft_consensus.cc:374] T 08e3096cdac34940a3ab0529cb76c3c7 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:20.175030 14587 raft_consensus.cc:729] T 08e3096cdac34940a3ab0529cb76c3c7 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:20.175145 14587 consensus_queue.cc:260] T 08e3096cdac34940a3ab0529cb76c3c7 P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:20.175230 14589 tablet_bootstrap.cc:654] T e8e52f0e9d3340b2bd468e2c7d8bc921 P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:20.175432 14587 ts_tablet_manager.cc:1398] T 08e3096cdac34940a3ab0529cb76c3c7 P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:20.175590 14587 tablet_bootstrap.cc:492] T e8e52f0e9d3340b2bd468e2c7d8bc921 P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:20.174618 14586 raft_consensus.cc:348] T 973a44e737844ffb99dd3e168fcdc1d5 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:20.175995 14586 raft_consensus.cc:374] T 973a44e737844ffb99dd3e168fcdc1d5 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:20.176039 14586 raft_consensus.cc:729] T 973a44e737844ffb99dd3e168fcdc1d5 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:20.176191 14586 consensus_queue.cc:260] T 973a44e737844ffb99dd3e168fcdc1d5 P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:20.176335 14589 tablet_bootstrap.cc:492] T e8e52f0e9d3340b2bd468e2c7d8bc921 P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:20.176411 14589 ts_tablet_manager.cc:1367] T e8e52f0e9d3340b2bd468e2c7d8bc921 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:56:20.176455 14587 tablet_bootstrap.cc:654] T e8e52f0e9d3340b2bd468e2c7d8bc921 P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:20.176929 14589 raft_consensus.cc:348] T e8e52f0e9d3340b2bd468e2c7d8bc921 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:20.177042 14589 raft_consensus.cc:374] T e8e52f0e9d3340b2bd468e2c7d8bc921 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:20.177064 14586 ts_tablet_manager.cc:1398] T 973a44e737844ffb99dd3e168fcdc1d5 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:56:20.177075 14589 raft_consensus.cc:729] T e8e52f0e9d3340b2bd468e2c7d8bc921 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:20.177213 14586 tablet_bootstrap.cc:492] T 08e3096cdac34940a3ab0529cb76c3c7 P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:20.177201 14589 consensus_queue.cc:260] T e8e52f0e9d3340b2bd468e2c7d8bc921 P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:20.177460 14589 ts_tablet_manager.cc:1398] T e8e52f0e9d3340b2bd468e2c7d8bc921 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:20.178483 14586 tablet_bootstrap.cc:654] T 08e3096cdac34940a3ab0529cb76c3c7 P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:20.178691 14587 tablet_bootstrap.cc:492] T e8e52f0e9d3340b2bd468e2c7d8bc921 P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:20.178768 14587 ts_tablet_manager.cc:1367] T e8e52f0e9d3340b2bd468e2c7d8bc921 P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 13:56:20.179172 14587 raft_consensus.cc:348] T e8e52f0e9d3340b2bd468e2c7d8bc921 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:20.179270 14587 raft_consensus.cc:374] T e8e52f0e9d3340b2bd468e2c7d8bc921 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:20.179302 14587 raft_consensus.cc:729] T e8e52f0e9d3340b2bd468e2c7d8bc921 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:20.179435 14587 consensus_queue.cc:260] T e8e52f0e9d3340b2bd468e2c7d8bc921 P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:20.179700 14587 ts_tablet_manager.cc:1398] T e8e52f0e9d3340b2bd468e2c7d8bc921 P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:20.181226 14586 tablet_bootstrap.cc:492] T 08e3096cdac34940a3ab0529cb76c3c7 P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:20.181313 14586 ts_tablet_manager.cc:1367] T 08e3096cdac34940a3ab0529cb76c3c7 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 13:56:20.181681 14586 raft_consensus.cc:348] T 08e3096cdac34940a3ab0529cb76c3c7 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:20.181768 14586 raft_consensus.cc:374] T 08e3096cdac34940a3ab0529cb76c3c7 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:20.181795 14586 raft_consensus.cc:729] T 08e3096cdac34940a3ab0529cb76c3c7 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:20.181919 14586 consensus_queue.cc:260] T 08e3096cdac34940a3ab0529cb76c3c7 P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:20.182157 14586 ts_tablet_manager.cc:1398] T 08e3096cdac34940a3ab0529cb76c3c7 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:20.189673 14230 raft_consensus.cc:482] T e8e52f0e9d3340b2bd468e2c7d8bc921 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:20.189814 14230 raft_consensus.cc:504] T e8e52f0e9d3340b2bd468e2c7d8bc921 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:20.190223 14230 leader_election.cc:288] T e8e52f0e9d3340b2bd468e2c7d8bc921 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279), 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:20.190601 13314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8e52f0e9d3340b2bd468e2c7d8bc921" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703" is_pre_election: true
I20240503 13:56:20.190603 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8e52f0e9d3340b2bd468e2c7d8bc921" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
I20240503 13:56:20.190739 13314 raft_consensus.cc:2454] T e8e52f0e9d3340b2bd468e2c7d8bc921 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 0.
I20240503 13:56:20.190757 13445 raft_consensus.cc:2454] T e8e52f0e9d3340b2bd468e2c7d8bc921 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 0.
I20240503 13:56:20.191007 12987 leader_election.cc:302] T e8e52f0e9d3340b2bd468e2c7d8bc921 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 7690ee79577c45e3a2c8f8b552c0bff8; no voters:
I20240503 13:56:20.191172 14230 raft_consensus.cc:2781] T e8e52f0e9d3340b2bd468e2c7d8bc921 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:20.191231 14230 raft_consensus.cc:482] T e8e52f0e9d3340b2bd468e2c7d8bc921 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:20.191257 14230 raft_consensus.cc:3037] T e8e52f0e9d3340b2bd468e2c7d8bc921 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:20.192091 14230 raft_consensus.cc:504] T e8e52f0e9d3340b2bd468e2c7d8bc921 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:20.192432 14230 leader_election.cc:288] T e8e52f0e9d3340b2bd468e2c7d8bc921 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 election: Requested vote from peers ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279), 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:20.192754 13314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8e52f0e9d3340b2bd468e2c7d8bc921" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703"
I20240503 13:56:20.192823 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8e52f0e9d3340b2bd468e2c7d8bc921" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
I20240503 13:56:20.192869 13314 raft_consensus.cc:3037] T e8e52f0e9d3340b2bd468e2c7d8bc921 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:20.192938 13445 raft_consensus.cc:3037] T e8e52f0e9d3340b2bd468e2c7d8bc921 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:20.193694 13314 raft_consensus.cc:2454] T e8e52f0e9d3340b2bd468e2c7d8bc921 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 1.
I20240503 13:56:20.193967 12990 leader_election.cc:302] T e8e52f0e9d3340b2bd468e2c7d8bc921 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:20.194008 13445 raft_consensus.cc:2454] T e8e52f0e9d3340b2bd468e2c7d8bc921 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 1.
I20240503 13:56:20.194128 14230 raft_consensus.cc:2781] T e8e52f0e9d3340b2bd468e2c7d8bc921 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:20.194202 14230 raft_consensus.cc:686] T e8e52f0e9d3340b2bd468e2c7d8bc921 P 60ba3ffe3b7746469834cd91c84be14e [term 1 LEADER]: Becoming Leader. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Running, Role: LEADER
I20240503 13:56:20.194339 14230 consensus_queue.cc:237] T e8e52f0e9d3340b2bd468e2c7d8bc921 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:20.196236 12911 catalog_manager.cc:5484] T e8e52f0e9d3340b2bd468e2c7d8bc921 P 60ba3ffe3b7746469834cd91c84be14e reported cstate change: term changed from 0 to 1, leader changed from <none> to 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65). New cstate: current_term: 1 leader_uuid: "60ba3ffe3b7746469834cd91c84be14e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } }
I20240503 13:56:20.266120 14580 raft_consensus.cc:482] T 973a44e737844ffb99dd3e168fcdc1d5 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:20.266320 14580 raft_consensus.cc:504] T 973a44e737844ffb99dd3e168fcdc1d5 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:20.266803 14580 leader_election.cc:288] T 973a44e737844ffb99dd3e168fcdc1d5 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:20.267177 13053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "973a44e737844ffb99dd3e168fcdc1d5" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e" is_pre_election: true
I20240503 13:56:20.267213 13184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "973a44e737844ffb99dd3e168fcdc1d5" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770" is_pre_election: true
I20240503 13:56:20.267309 13053 raft_consensus.cc:2454] T 973a44e737844ffb99dd3e168fcdc1d5 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 0.
I20240503 13:56:20.267385 13184 raft_consensus.cc:2454] T 973a44e737844ffb99dd3e168fcdc1d5 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 0.
I20240503 13:56:20.267601 13250 leader_election.cc:302] T 973a44e737844ffb99dd3e168fcdc1d5 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:20.267747 14580 raft_consensus.cc:2781] T 973a44e737844ffb99dd3e168fcdc1d5 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:20.267805 14580 raft_consensus.cc:482] T 973a44e737844ffb99dd3e168fcdc1d5 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:20.267838 14580 raft_consensus.cc:3037] T 973a44e737844ffb99dd3e168fcdc1d5 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:20.268663 14580 raft_consensus.cc:504] T 973a44e737844ffb99dd3e168fcdc1d5 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:20.268956 14580 leader_election.cc:288] T 973a44e737844ffb99dd3e168fcdc1d5 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: Requested vote from peers 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:20.269246 13053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "973a44e737844ffb99dd3e168fcdc1d5" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e"
I20240503 13:56:20.269351 13053 raft_consensus.cc:3037] T 973a44e737844ffb99dd3e168fcdc1d5 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:20.269575 13184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "973a44e737844ffb99dd3e168fcdc1d5" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770"
I20240503 13:56:20.269690 13184 raft_consensus.cc:3037] T 973a44e737844ffb99dd3e168fcdc1d5 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:20.270114 13053 raft_consensus.cc:2454] T 973a44e737844ffb99dd3e168fcdc1d5 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 1.
I20240503 13:56:20.270352 13250 leader_election.cc:302] T 973a44e737844ffb99dd3e168fcdc1d5 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:20.270416 13184 raft_consensus.cc:2454] T 973a44e737844ffb99dd3e168fcdc1d5 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 1.
I20240503 13:56:20.270503 14438 raft_consensus.cc:2781] T 973a44e737844ffb99dd3e168fcdc1d5 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:20.270576 14438 raft_consensus.cc:686] T 973a44e737844ffb99dd3e168fcdc1d5 P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Becoming Leader. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Running, Role: LEADER
I20240503 13:56:20.270701 14438 consensus_queue.cc:237] T 973a44e737844ffb99dd3e168fcdc1d5 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:20.272522 12911 catalog_manager.cc:5484] T 973a44e737844ffb99dd3e168fcdc1d5 P ba0194098aa84449ae17e8a0d4c43703 reported cstate change: term changed from 0 to 1, leader changed from <none> to ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67). New cstate: current_term: 1 leader_uuid: "ba0194098aa84449ae17e8a0d4c43703" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:20.462124 14572 raft_consensus.cc:482] T 08e3096cdac34940a3ab0529cb76c3c7 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:20.462256 14572 raft_consensus.cc:504] T 08e3096cdac34940a3ab0529cb76c3c7 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:20.462675 14572 leader_election.cc:288] T 08e3096cdac34940a3ab0529cb76c3c7 P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695), 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:20.463215 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08e3096cdac34940a3ab0529cb76c3c7" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
I20240503 13:56:20.463215 13053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08e3096cdac34940a3ab0529cb76c3c7" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e" is_pre_election: true
I20240503 13:56:20.463367 13053 raft_consensus.cc:2454] T 08e3096cdac34940a3ab0529cb76c3c7 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 0.
I20240503 13:56:20.463367 13445 raft_consensus.cc:2454] T 08e3096cdac34940a3ab0529cb76c3c7 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 0.
I20240503 13:56:20.463716 13118 leader_election.cc:302] T 08e3096cdac34940a3ab0529cb76c3c7 P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, 8ccf5a4ac450414c9d3579bd5800b770; no voters:
I20240503 13:56:20.463910 14572 raft_consensus.cc:2781] T 08e3096cdac34940a3ab0529cb76c3c7 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:20.463989 14572 raft_consensus.cc:482] T 08e3096cdac34940a3ab0529cb76c3c7 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:20.464020 14572 raft_consensus.cc:3037] T 08e3096cdac34940a3ab0529cb76c3c7 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:20.464942 14572 raft_consensus.cc:504] T 08e3096cdac34940a3ab0529cb76c3c7 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:20.465297 14572 leader_election.cc:288] T 08e3096cdac34940a3ab0529cb76c3c7 P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 election: Requested vote from peers 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695), 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:20.465669 13053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08e3096cdac34940a3ab0529cb76c3c7" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e"
I20240503 13:56:20.465740 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08e3096cdac34940a3ab0529cb76c3c7" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
I20240503 13:56:20.465792 13053 raft_consensus.cc:3037] T 08e3096cdac34940a3ab0529cb76c3c7 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:20.465915 13445 raft_consensus.cc:3037] T 08e3096cdac34940a3ab0529cb76c3c7 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:20.466612 13053 raft_consensus.cc:2454] T 08e3096cdac34940a3ab0529cb76c3c7 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 1.
I20240503 13:56:20.466667 13445 raft_consensus.cc:2454] T 08e3096cdac34940a3ab0529cb76c3c7 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 1.
I20240503 13:56:20.466909 13119 leader_election.cc:302] T 08e3096cdac34940a3ab0529cb76c3c7 P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 8ccf5a4ac450414c9d3579bd5800b770; no voters:
I20240503 13:56:20.467080 14572 raft_consensus.cc:2781] T 08e3096cdac34940a3ab0529cb76c3c7 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:20.467167 14572 raft_consensus.cc:686] T 08e3096cdac34940a3ab0529cb76c3c7 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 LEADER]: Becoming Leader. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Running, Role: LEADER
I20240503 13:56:20.467497 14572 consensus_queue.cc:237] T 08e3096cdac34940a3ab0529cb76c3c7 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:20.469201 12911 catalog_manager.cc:5484] T 08e3096cdac34940a3ab0529cb76c3c7 P 8ccf5a4ac450414c9d3579bd5800b770 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66). New cstate: current_term: 1 leader_uuid: "8ccf5a4ac450414c9d3579bd5800b770" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } }
W20240503 13:56:20.638655 14602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:56:20.638913 14602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:56:20.642087 14602 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240503 13:56:20.714802 14438 consensus_queue.cc:1035] T 973a44e737844ffb99dd3e168fcdc1d5 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:20.722909 13314 raft_consensus.cc:1261] T e8e52f0e9d3340b2bd468e2c7d8bc921 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Refusing update from remote peer 60ba3ffe3b7746469834cd91c84be14e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:20.723093 13052 raft_consensus.cc:1261] T 08e3096cdac34940a3ab0529cb76c3c7 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Refusing update from remote peer 8ccf5a4ac450414c9d3579bd5800b770: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:20.723498 14641 consensus_queue.cc:1035] T e8e52f0e9d3340b2bd468e2c7d8bc921 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:20.723563 13445 raft_consensus.cc:1261] T e8e52f0e9d3340b2bd468e2c7d8bc921 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer 60ba3ffe3b7746469834cd91c84be14e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:20.724992 14572 consensus_queue.cc:1035] T 08e3096cdac34940a3ab0529cb76c3c7 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:20.725490 14483 consensus_queue.cc:1035] T e8e52f0e9d3340b2bd468e2c7d8bc921 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:20.726716 13445 raft_consensus.cc:1261] T 08e3096cdac34940a3ab0529cb76c3c7 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer 8ccf5a4ac450414c9d3579bd5800b770: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:20.733147 14572 consensus_queue.cc:1035] T 08e3096cdac34940a3ab0529cb76c3c7 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:20.733211 13184 raft_consensus.cc:1261] T 973a44e737844ffb99dd3e168fcdc1d5 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:20.734098 14580 consensus_queue.cc:1035] T 973a44e737844ffb99dd3e168fcdc1d5 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 107.015 ms
time per row: 0.0535077 ms
I20240503 13:56:20.791432 12912 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:56:20.796963 13426 tablet_service.cc:1467] Processing CreateTablet for tablet b82134ddae61479eb2ced348d7558799 (DEFAULT_TABLE table=rebalance_test_table_9 [id=2fd11e7ec03a4f6891fc18482c1728a6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:20.796938 13295 tablet_service.cc:1467] Processing CreateTablet for tablet b82134ddae61479eb2ced348d7558799 (DEFAULT_TABLE table=rebalance_test_table_9 [id=2fd11e7ec03a4f6891fc18482c1728a6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:20.797262 13426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b82134ddae61479eb2ced348d7558799. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:20.797266 13295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b82134ddae61479eb2ced348d7558799. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:20.797886 13164 tablet_service.cc:1467] Processing CreateTablet for tablet b291e3dd6a7a454ca4d736eff8dd3a41 (DEFAULT_TABLE table=rebalance_test_table_9 [id=2fd11e7ec03a4f6891fc18482c1728a6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:20.797899 13163 tablet_service.cc:1467] Processing CreateTablet for tablet d50d05d432e84d5eb0e54d1a68cdf158 (DEFAULT_TABLE table=rebalance_test_table_9 [id=2fd11e7ec03a4f6891fc18482c1728a6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:20.798202 13164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b291e3dd6a7a454ca4d736eff8dd3a41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:20.798327 13163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d50d05d432e84d5eb0e54d1a68cdf158. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:20.799654 13165 tablet_service.cc:1467] Processing CreateTablet for tablet b82134ddae61479eb2ced348d7558799 (DEFAULT_TABLE table=rebalance_test_table_9 [id=2fd11e7ec03a4f6891fc18482c1728a6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:20.799870 13165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b82134ddae61479eb2ced348d7558799. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:20.800408 13426 tablet_service.cc:1467] Processing CreateTablet for tablet b291e3dd6a7a454ca4d736eff8dd3a41 (DEFAULT_TABLE table=rebalance_test_table_9 [id=2fd11e7ec03a4f6891fc18482c1728a6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:20.800663 14669 tablet_bootstrap.cc:492] T b291e3dd6a7a454ca4d736eff8dd3a41 P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:20.800675 13426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b291e3dd6a7a454ca4d736eff8dd3a41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:20.801537 14669 tablet_bootstrap.cc:654] T b291e3dd6a7a454ca4d736eff8dd3a41 P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:20.802098 13032 tablet_service.cc:1467] Processing CreateTablet for tablet d50d05d432e84d5eb0e54d1a68cdf158 (DEFAULT_TABLE table=rebalance_test_table_9 [id=2fd11e7ec03a4f6891fc18482c1728a6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:20.802361 13032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d50d05d432e84d5eb0e54d1a68cdf158. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:20.802377 14667 tablet_bootstrap.cc:492] T b82134ddae61479eb2ced348d7558799 P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:20.802661 14669 tablet_bootstrap.cc:492] T b291e3dd6a7a454ca4d736eff8dd3a41 P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:20.802735 14669 ts_tablet_manager.cc:1367] T b291e3dd6a7a454ca4d736eff8dd3a41 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:20.803012 14667 tablet_bootstrap.cc:654] T b82134ddae61479eb2ced348d7558799 P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:20.803145 14669 raft_consensus.cc:348] T b291e3dd6a7a454ca4d736eff8dd3a41 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:20.803246 14669 raft_consensus.cc:374] T b291e3dd6a7a454ca4d736eff8dd3a41 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:20.803277 14669 raft_consensus.cc:729] T b291e3dd6a7a454ca4d736eff8dd3a41 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:20.803404 14669 consensus_queue.cc:260] T b291e3dd6a7a454ca4d736eff8dd3a41 P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:20.803686 14669 ts_tablet_manager.cc:1398] T b291e3dd6a7a454ca4d736eff8dd3a41 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:20.804152 14667 tablet_bootstrap.cc:492] T b82134ddae61479eb2ced348d7558799 P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:20.804219 14667 ts_tablet_manager.cc:1367] T b82134ddae61479eb2ced348d7558799 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:20.804694 14667 raft_consensus.cc:348] T b82134ddae61479eb2ced348d7558799 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:20.804788 14667 raft_consensus.cc:374] T b82134ddae61479eb2ced348d7558799 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:20.804791 14669 tablet_bootstrap.cc:492] T d50d05d432e84d5eb0e54d1a68cdf158 P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:20.804818 14667 raft_consensus.cc:729] T b82134ddae61479eb2ced348d7558799 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:20.804946 14667 consensus_queue.cc:260] T b82134ddae61479eb2ced348d7558799 P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:20.805205 13294 tablet_service.cc:1467] Processing CreateTablet for tablet d50d05d432e84d5eb0e54d1a68cdf158 (DEFAULT_TABLE table=rebalance_test_table_9 [id=2fd11e7ec03a4f6891fc18482c1728a6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:20.805275 14667 ts_tablet_manager.cc:1398] T b82134ddae61479eb2ced348d7558799 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:20.805418 14667 tablet_bootstrap.cc:492] T b291e3dd6a7a454ca4d736eff8dd3a41 P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:20.805457 13294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d50d05d432e84d5eb0e54d1a68cdf158. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:20.805887 14669 tablet_bootstrap.cc:654] T d50d05d432e84d5eb0e54d1a68cdf158 P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:20.806203 14667 tablet_bootstrap.cc:654] T b291e3dd6a7a454ca4d736eff8dd3a41 P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:20.806514 14672 tablet_bootstrap.cc:492] T d50d05d432e84d5eb0e54d1a68cdf158 P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:20.808264 14667 tablet_bootstrap.cc:492] T b291e3dd6a7a454ca4d736eff8dd3a41 P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:20.808302 14672 tablet_bootstrap.cc:654] T d50d05d432e84d5eb0e54d1a68cdf158 P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:20.808339 14667 ts_tablet_manager.cc:1367] T b291e3dd6a7a454ca4d736eff8dd3a41 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:56:20.808542 13295 tablet_service.cc:1467] Processing CreateTablet for tablet b291e3dd6a7a454ca4d736eff8dd3a41 (DEFAULT_TABLE table=rebalance_test_table_9 [id=2fd11e7ec03a4f6891fc18482c1728a6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:20.808688 14668 tablet_bootstrap.cc:492] T b82134ddae61479eb2ced348d7558799 P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:20.808780 13295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b291e3dd6a7a454ca4d736eff8dd3a41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:20.808866 14667 raft_consensus.cc:348] T b291e3dd6a7a454ca4d736eff8dd3a41 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:20.808957 14667 raft_consensus.cc:374] T b291e3dd6a7a454ca4d736eff8dd3a41 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:20.808987 14667 raft_consensus.cc:729] T b291e3dd6a7a454ca4d736eff8dd3a41 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:20.809119 14667 consensus_queue.cc:260] T b291e3dd6a7a454ca4d736eff8dd3a41 P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:20.809329 14669 tablet_bootstrap.cc:492] T d50d05d432e84d5eb0e54d1a68cdf158 P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:20.809399 14669 ts_tablet_manager.cc:1367] T d50d05d432e84d5eb0e54d1a68cdf158 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 13:56:20.809780 14668 tablet_bootstrap.cc:654] T b82134ddae61479eb2ced348d7558799 P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:20.809824 14669 raft_consensus.cc:348] T d50d05d432e84d5eb0e54d1a68cdf158 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:20.810155 14669 raft_consensus.cc:374] T d50d05d432e84d5eb0e54d1a68cdf158 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:20.810200 14669 raft_consensus.cc:729] T d50d05d432e84d5eb0e54d1a68cdf158 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:20.810343 14669 consensus_queue.cc:260] T d50d05d432e84d5eb0e54d1a68cdf158 P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:20.811167 14667 ts_tablet_manager.cc:1398] T b291e3dd6a7a454ca4d736eff8dd3a41 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:56:20.811929 14668 tablet_bootstrap.cc:492] T b82134ddae61479eb2ced348d7558799 P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:20.811879 14669 ts_tablet_manager.cc:1398] T d50d05d432e84d5eb0e54d1a68cdf158 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:20.812016 14668 ts_tablet_manager.cc:1367] T b82134ddae61479eb2ced348d7558799 P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:56:20.812408 14668 raft_consensus.cc:348] T b82134ddae61479eb2ced348d7558799 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:20.812541 14668 raft_consensus.cc:374] T b82134ddae61479eb2ced348d7558799 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:20.812577 14668 raft_consensus.cc:729] T b82134ddae61479eb2ced348d7558799 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:20.812592 14669 tablet_bootstrap.cc:492] T b82134ddae61479eb2ced348d7558799 P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:20.812693 14668 consensus_queue.cc:260] T b82134ddae61479eb2ced348d7558799 P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:20.812947 14668 ts_tablet_manager.cc:1398] T b82134ddae61479eb2ced348d7558799 P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:20.813100 14668 tablet_bootstrap.cc:492] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:20.813499 14669 tablet_bootstrap.cc:654] T b82134ddae61479eb2ced348d7558799 P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:20.814126 14668 tablet_bootstrap.cc:654] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:20.814761 14672 tablet_bootstrap.cc:492] T d50d05d432e84d5eb0e54d1a68cdf158 P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:20.815212 14669 tablet_bootstrap.cc:492] T b82134ddae61479eb2ced348d7558799 P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:20.815285 14668 tablet_bootstrap.cc:492] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:20.815363 14668 ts_tablet_manager.cc:1367] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:56:20.815364 14669 ts_tablet_manager.cc:1367] T b82134ddae61479eb2ced348d7558799 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:56:20.815685 14668 raft_consensus.cc:348] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:20.815754 14668 raft_consensus.cc:374] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:20.815774 14668 raft_consensus.cc:729] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:20.815735 14669 raft_consensus.cc:348] T b82134ddae61479eb2ced348d7558799 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:20.815817 14669 raft_consensus.cc:374] T b82134ddae61479eb2ced348d7558799 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:20.815846 14669 raft_consensus.cc:729] T b82134ddae61479eb2ced348d7558799 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:20.815858 14668 consensus_queue.cc:260] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:20.815970 14669 consensus_queue.cc:260] T b82134ddae61479eb2ced348d7558799 P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:20.816068 14668 ts_tablet_manager.cc:1398] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:20.816210 14668 tablet_bootstrap.cc:492] T b291e3dd6a7a454ca4d736eff8dd3a41 P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:20.816704 14672 ts_tablet_manager.cc:1367] T d50d05d432e84d5eb0e54d1a68cdf158 P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 13:56:20.816962 14668 tablet_bootstrap.cc:654] T b291e3dd6a7a454ca4d736eff8dd3a41 P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:20.817209 14672 raft_consensus.cc:348] T d50d05d432e84d5eb0e54d1a68cdf158 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:20.817342 14672 raft_consensus.cc:374] T d50d05d432e84d5eb0e54d1a68cdf158 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:20.817380 14672 raft_consensus.cc:729] T d50d05d432e84d5eb0e54d1a68cdf158 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:20.817651 14669 ts_tablet_manager.cc:1398] T b82134ddae61479eb2ced348d7558799 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:20.817857 14672 consensus_queue.cc:260] T d50d05d432e84d5eb0e54d1a68cdf158 P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:20.818185 14672 ts_tablet_manager.cc:1398] T d50d05d432e84d5eb0e54d1a68cdf158 P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:20.820227 14668 tablet_bootstrap.cc:492] T b291e3dd6a7a454ca4d736eff8dd3a41 P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:20.820322 14668 ts_tablet_manager.cc:1367] T b291e3dd6a7a454ca4d736eff8dd3a41 P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 13:56:20.820835 14668 raft_consensus.cc:348] T b291e3dd6a7a454ca4d736eff8dd3a41 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:20.820951 14668 raft_consensus.cc:374] T b291e3dd6a7a454ca4d736eff8dd3a41 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:20.820997 14668 raft_consensus.cc:729] T b291e3dd6a7a454ca4d736eff8dd3a41 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:20.821123 14668 consensus_queue.cc:260] T b291e3dd6a7a454ca4d736eff8dd3a41 P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:20.821420 14668 ts_tablet_manager.cc:1398] T b291e3dd6a7a454ca4d736eff8dd3a41 P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:20.897845 14597 raft_consensus.cc:482] T b291e3dd6a7a454ca4d736eff8dd3a41 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:20.897979 14597 raft_consensus.cc:504] T b291e3dd6a7a454ca4d736eff8dd3a41 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:20.898488 14597 leader_election.cc:288] T b291e3dd6a7a454ca4d736eff8dd3a41 P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:20.898978 13314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b291e3dd6a7a454ca4d736eff8dd3a41" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703" is_pre_election: true
I20240503 13:56:20.898978 13184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b291e3dd6a7a454ca4d736eff8dd3a41" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770" is_pre_election: true
I20240503 13:56:20.899127 13314 raft_consensus.cc:2454] T b291e3dd6a7a454ca4d736eff8dd3a41 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 0.
I20240503 13:56:20.899127 13184 raft_consensus.cc:2454] T b291e3dd6a7a454ca4d736eff8dd3a41 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 0.
I20240503 13:56:20.899430 13383 leader_election.cc:302] T b291e3dd6a7a454ca4d736eff8dd3a41 P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:20.899595 14597 raft_consensus.cc:2781] T b291e3dd6a7a454ca4d736eff8dd3a41 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:20.899669 14597 raft_consensus.cc:482] T b291e3dd6a7a454ca4d736eff8dd3a41 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:20.899717 14597 raft_consensus.cc:3037] T b291e3dd6a7a454ca4d736eff8dd3a41 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:20.900616 14597 raft_consensus.cc:504] T b291e3dd6a7a454ca4d736eff8dd3a41 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:20.900918 14597 leader_election.cc:288] T b291e3dd6a7a454ca4d736eff8dd3a41 P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 election: Requested vote from peers ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:20.901268 13314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b291e3dd6a7a454ca4d736eff8dd3a41" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703"
I20240503 13:56:20.901376 13314 raft_consensus.cc:3037] T b291e3dd6a7a454ca4d736eff8dd3a41 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:20.901327 13184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b291e3dd6a7a454ca4d736eff8dd3a41" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770"
I20240503 13:56:20.901464 13184 raft_consensus.cc:3037] T b291e3dd6a7a454ca4d736eff8dd3a41 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:20.902117 13314 raft_consensus.cc:2454] T b291e3dd6a7a454ca4d736eff8dd3a41 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 1.
I20240503 13:56:20.902299 13184 raft_consensus.cc:2454] T b291e3dd6a7a454ca4d736eff8dd3a41 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 1.
I20240503 13:56:20.902405 13383 leader_election.cc:302] T b291e3dd6a7a454ca4d736eff8dd3a41 P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:20.902560 14597 raft_consensus.cc:2781] T b291e3dd6a7a454ca4d736eff8dd3a41 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:20.902771 14597 raft_consensus.cc:686] T b291e3dd6a7a454ca4d736eff8dd3a41 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 LEADER]: Becoming Leader. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Running, Role: LEADER
I20240503 13:56:20.902925 14597 consensus_queue.cc:237] T b291e3dd6a7a454ca4d736eff8dd3a41 P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:20.904742 12912 catalog_manager.cc:5484] T b291e3dd6a7a454ca4d736eff8dd3a41 P 7690ee79577c45e3a2c8f8b552c0bff8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68). New cstate: current_term: 1 leader_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } }
I20240503 13:56:20.965400 14572 raft_consensus.cc:482] T b82134ddae61479eb2ced348d7558799 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:20.965553 14572 raft_consensus.cc:504] T b82134ddae61479eb2ced348d7558799 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:20.966051 14572 leader_election.cc:288] T b82134ddae61479eb2ced348d7558799 P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279), 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:20.966511 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b82134ddae61479eb2ced348d7558799" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
I20240503 13:56:20.966503 13314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b82134ddae61479eb2ced348d7558799" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703" is_pre_election: true
I20240503 13:56:20.966655 13314 raft_consensus.cc:2454] T b82134ddae61479eb2ced348d7558799 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 0.
I20240503 13:56:20.966655 13446 raft_consensus.cc:2454] T b82134ddae61479eb2ced348d7558799 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 0.
I20240503 13:56:20.966972 13118 leader_election.cc:302] T b82134ddae61479eb2ced348d7558799 P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, 8ccf5a4ac450414c9d3579bd5800b770; no voters:
I20240503 13:56:20.967162 14572 raft_consensus.cc:2781] T b82134ddae61479eb2ced348d7558799 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:20.967218 14572 raft_consensus.cc:482] T b82134ddae61479eb2ced348d7558799 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:20.967245 14572 raft_consensus.cc:3037] T b82134ddae61479eb2ced348d7558799 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:20.968091 14572 raft_consensus.cc:504] T b82134ddae61479eb2ced348d7558799 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:20.968415 14572 leader_election.cc:288] T b82134ddae61479eb2ced348d7558799 P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 election: Requested vote from peers ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279), 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:20.968768 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b82134ddae61479eb2ced348d7558799" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
I20240503 13:56:20.968923 13446 raft_consensus.cc:3037] T b82134ddae61479eb2ced348d7558799 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:20.969087 13314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b82134ddae61479eb2ced348d7558799" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703"
I20240503 13:56:20.969205 13314 raft_consensus.cc:3037] T b82134ddae61479eb2ced348d7558799 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:20.969892 13446 raft_consensus.cc:2454] T b82134ddae61479eb2ced348d7558799 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 1.
I20240503 13:56:20.970062 13314 raft_consensus.cc:2454] T b82134ddae61479eb2ced348d7558799 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 1.
I20240503 13:56:20.970202 13118 leader_election.cc:302] T b82134ddae61479eb2ced348d7558799 P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, 8ccf5a4ac450414c9d3579bd5800b770; no voters:
I20240503 13:56:20.970361 14572 raft_consensus.cc:2781] T b82134ddae61479eb2ced348d7558799 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:20.970470 14572 raft_consensus.cc:686] T b82134ddae61479eb2ced348d7558799 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 LEADER]: Becoming Leader. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Running, Role: LEADER
I20240503 13:56:20.970801 14572 consensus_queue.cc:237] T b82134ddae61479eb2ced348d7558799 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:20.972762 12912 catalog_manager.cc:5484] T b82134ddae61479eb2ced348d7558799 P 8ccf5a4ac450414c9d3579bd5800b770 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66). New cstate: current_term: 1 leader_uuid: "8ccf5a4ac450414c9d3579bd5800b770" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } }
I20240503 13:56:21.112735 14650 raft_consensus.cc:482] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:21.112861 14650 raft_consensus.cc:504] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:21.113303 14650 leader_election.cc:288] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:21.113657 13052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d50d05d432e84d5eb0e54d1a68cdf158" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e" is_pre_election: true
I20240503 13:56:21.113657 13184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d50d05d432e84d5eb0e54d1a68cdf158" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770" is_pre_election: true
I20240503 13:56:21.113799 13052 raft_consensus.cc:2454] T d50d05d432e84d5eb0e54d1a68cdf158 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 0.
I20240503 13:56:21.113799 13184 raft_consensus.cc:2454] T d50d05d432e84d5eb0e54d1a68cdf158 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 0.
I20240503 13:56:21.114109 13250 leader_election.cc:302] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 8ccf5a4ac450414c9d3579bd5800b770, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:21.114318 14650 raft_consensus.cc:2781] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:21.114444 14650 raft_consensus.cc:482] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:21.114527 14650 raft_consensus.cc:3037] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:21.115418 14650 raft_consensus.cc:504] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:21.115775 14650 leader_election.cc:288] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: Requested vote from peers 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:21.116202 13053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d50d05d432e84d5eb0e54d1a68cdf158" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e"
I20240503 13:56:21.116230 13184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d50d05d432e84d5eb0e54d1a68cdf158" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770"
I20240503 13:56:21.116328 13053 raft_consensus.cc:3037] T d50d05d432e84d5eb0e54d1a68cdf158 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:21.116348 13184 raft_consensus.cc:3037] T d50d05d432e84d5eb0e54d1a68cdf158 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:21.117341 13053 raft_consensus.cc:2454] T d50d05d432e84d5eb0e54d1a68cdf158 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 1.
I20240503 13:56:21.117360 13184 raft_consensus.cc:2454] T d50d05d432e84d5eb0e54d1a68cdf158 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 1.
I20240503 13:56:21.117756 13250 leader_election.cc:302] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:21.117931 14650 raft_consensus.cc:2781] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:21.118022 14650 raft_consensus.cc:686] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Becoming Leader. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Running, Role: LEADER
I20240503 13:56:21.118161 14650 consensus_queue.cc:237] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:21.120193 12912 catalog_manager.cc:5484] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 reported cstate change: term changed from 0 to 1, leader changed from <none> to ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67). New cstate: current_term: 1 leader_uuid: "ba0194098aa84449ae17e8a0d4c43703" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } }
W20240503 13:56:21.271553 14684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:56:21.271817 14684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:56:21.275012 14684 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240503 13:56:21.375519 13184 raft_consensus.cc:1261] T b291e3dd6a7a454ca4d736eff8dd3a41 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Refusing update from remote peer 7690ee79577c45e3a2c8f8b552c0bff8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:21.375960 13315 raft_consensus.cc:1261] T b291e3dd6a7a454ca4d736eff8dd3a41 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Refusing update from remote peer 7690ee79577c45e3a2c8f8b552c0bff8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:21.375960 14597 consensus_queue.cc:1035] T b291e3dd6a7a454ca4d736eff8dd3a41 P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:21.377240 14597 consensus_queue.cc:1035] T b291e3dd6a7a454ca4d736eff8dd3a41 P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:21.382477 13314 raft_consensus.cc:1261] T b82134ddae61479eb2ced348d7558799 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Refusing update from remote peer 8ccf5a4ac450414c9d3579bd5800b770: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:21.383600 13446 raft_consensus.cc:1261] T b82134ddae61479eb2ced348d7558799 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer 8ccf5a4ac450414c9d3579bd5800b770: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:21.383713 14253 consensus_queue.cc:1035] T b82134ddae61479eb2ced348d7558799 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:21.384505 14253 consensus_queue.cc:1035] T b82134ddae61479eb2ced348d7558799 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:21.389287 13053 raft_consensus.cc:1261] T d50d05d432e84d5eb0e54d1a68cdf158 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:21.389776 14438 consensus_queue.cc:1035] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:21.392292 13184 raft_consensus.cc:1261] T d50d05d432e84d5eb0e54d1a68cdf158 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:21.392921 14438 consensus_queue.cc:1035] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 112.553 ms
time per row: 0.0562763 ms
I20240503 13:56:21.442183 12908 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalance_test_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:56:21.447695 13032 tablet_service.cc:1467] Processing CreateTablet for tablet 67c4b85c17ab4e59be2d6223d993b48c (DEFAULT_TABLE table=rebalance_test_table_10 [id=0e569dec5058486fae2fb03b2b415c0f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:21.447769 13165 tablet_service.cc:1467] Processing CreateTablet for tablet 67c4b85c17ab4e59be2d6223d993b48c (DEFAULT_TABLE table=rebalance_test_table_10 [id=0e569dec5058486fae2fb03b2b415c0f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:21.448043 13032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67c4b85c17ab4e59be2d6223d993b48c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:21.448050 13165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67c4b85c17ab4e59be2d6223d993b48c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:21.448125 13033 tablet_service.cc:1467] Processing CreateTablet for tablet 3d480dd03b6a4d9fb9f64c861d52a720 (DEFAULT_TABLE table=rebalance_test_table_10 [id=0e569dec5058486fae2fb03b2b415c0f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:21.448427 13033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d480dd03b6a4d9fb9f64c861d52a720. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:21.448781 13295 tablet_service.cc:1467] Processing CreateTablet for tablet 3d480dd03b6a4d9fb9f64c861d52a720 (DEFAULT_TABLE table=rebalance_test_table_10 [id=0e569dec5058486fae2fb03b2b415c0f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:21.449024 13295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d480dd03b6a4d9fb9f64c861d52a720. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:21.450697 13165 tablet_service.cc:1467] Processing CreateTablet for tablet ed3d2ea6e2d74bc5ba60dbbbf2e7b41d (DEFAULT_TABLE table=rebalance_test_table_10 [id=0e569dec5058486fae2fb03b2b415c0f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:21.450750 14742 tablet_bootstrap.cc:492] T 67c4b85c17ab4e59be2d6223d993b48c P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:21.450764 13427 tablet_service.cc:1467] Processing CreateTablet for tablet ed3d2ea6e2d74bc5ba60dbbbf2e7b41d (DEFAULT_TABLE table=rebalance_test_table_10 [id=0e569dec5058486fae2fb03b2b415c0f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:21.450927 13165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed3d2ea6e2d74bc5ba60dbbbf2e7b41d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:21.450996 13427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed3d2ea6e2d74bc5ba60dbbbf2e7b41d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:21.451382 13426 tablet_service.cc:1467] Processing CreateTablet for tablet 67c4b85c17ab4e59be2d6223d993b48c (DEFAULT_TABLE table=rebalance_test_table_10 [id=0e569dec5058486fae2fb03b2b415c0f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:21.451606 13426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67c4b85c17ab4e59be2d6223d993b48c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:21.451637 14742 tablet_bootstrap.cc:654] T 67c4b85c17ab4e59be2d6223d993b48c P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:21.453042 14746 tablet_bootstrap.cc:492] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:21.453437 13425 tablet_service.cc:1467] Processing CreateTablet for tablet 3d480dd03b6a4d9fb9f64c861d52a720 (DEFAULT_TABLE table=rebalance_test_table_10 [id=0e569dec5058486fae2fb03b2b415c0f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:21.453563 14743 tablet_bootstrap.cc:492] T 67c4b85c17ab4e59be2d6223d993b48c P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:21.453665 13425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d480dd03b6a4d9fb9f64c861d52a720. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:21.453886 14746 tablet_bootstrap.cc:654] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:21.454434 14743 tablet_bootstrap.cc:654] T 67c4b85c17ab4e59be2d6223d993b48c P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:21.454943 14746 tablet_bootstrap.cc:492] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:21.455047 14746 ts_tablet_manager.cc:1367] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:21.455482 14746 raft_consensus.cc:348] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:21.455554 14743 tablet_bootstrap.cc:492] T 67c4b85c17ab4e59be2d6223d993b48c P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:21.455590 14746 raft_consensus.cc:374] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:21.455621 14746 raft_consensus.cc:729] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:21.455631 14743 ts_tablet_manager.cc:1367] T 67c4b85c17ab4e59be2d6223d993b48c P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:56:21.455734 14746 consensus_queue.cc:260] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:21.456007 14746 ts_tablet_manager.cc:1398] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:21.456140 14746 tablet_bootstrap.cc:492] T 67c4b85c17ab4e59be2d6223d993b48c P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:21.456259 13033 tablet_service.cc:1467] Processing CreateTablet for tablet ed3d2ea6e2d74bc5ba60dbbbf2e7b41d (DEFAULT_TABLE table=rebalance_test_table_10 [id=0e569dec5058486fae2fb03b2b415c0f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:21.456246 14743 raft_consensus.cc:348] T 67c4b85c17ab4e59be2d6223d993b48c P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:21.456347 14743 raft_consensus.cc:374] T 67c4b85c17ab4e59be2d6223d993b48c P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:21.456375 14743 raft_consensus.cc:729] T 67c4b85c17ab4e59be2d6223d993b48c P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:21.456511 14743 consensus_queue.cc:260] T 67c4b85c17ab4e59be2d6223d993b48c P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:21.456677 13033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed3d2ea6e2d74bc5ba60dbbbf2e7b41d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:21.456782 14743 ts_tablet_manager.cc:1398] T 67c4b85c17ab4e59be2d6223d993b48c P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:21.456934 14743 tablet_bootstrap.cc:492] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:21.456945 14744 tablet_bootstrap.cc:492] T 3d480dd03b6a4d9fb9f64c861d52a720 P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:21.456975 14746 tablet_bootstrap.cc:654] T 67c4b85c17ab4e59be2d6223d993b48c P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:21.457762 14743 tablet_bootstrap.cc:654] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:21.457769 14744 tablet_bootstrap.cc:654] T 3d480dd03b6a4d9fb9f64c861d52a720 P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:21.459161 14742 tablet_bootstrap.cc:492] T 67c4b85c17ab4e59be2d6223d993b48c P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:21.459172 14743 tablet_bootstrap.cc:492] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:21.459245 14743 ts_tablet_manager.cc:1367] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:21.459337 14742 ts_tablet_manager.cc:1367] T 67c4b85c17ab4e59be2d6223d993b48c P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 13:56:21.459674 14743 raft_consensus.cc:348] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:21.459784 14743 raft_consensus.cc:374] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:21.459821 14743 raft_consensus.cc:729] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:21.459879 14742 raft_consensus.cc:348] T 67c4b85c17ab4e59be2d6223d993b48c P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:21.460005 14742 raft_consensus.cc:374] T 67c4b85c17ab4e59be2d6223d993b48c P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:21.459972 14743 consensus_queue.cc:260] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:21.460057 14742 raft_consensus.cc:729] T 67c4b85c17ab4e59be2d6223d993b48c P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:21.460207 14742 consensus_queue.cc:260] T 67c4b85c17ab4e59be2d6223d993b48c P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:21.460988 14742 ts_tablet_manager.cc:1398] T 67c4b85c17ab4e59be2d6223d993b48c P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:21.461150 14742 tablet_bootstrap.cc:492] T 3d480dd03b6a4d9fb9f64c861d52a720 P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:21.461922 14742 tablet_bootstrap.cc:654] T 3d480dd03b6a4d9fb9f64c861d52a720 P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:21.460358 14743 ts_tablet_manager.cc:1398] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:21.462981 14744 tablet_bootstrap.cc:492] T 3d480dd03b6a4d9fb9f64c861d52a720 P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:21.463136 14744 ts_tablet_manager.cc:1367] T 3d480dd03b6a4d9fb9f64c861d52a720 P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 13:56:21.463682 14744 raft_consensus.cc:348] T 3d480dd03b6a4d9fb9f64c861d52a720 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:21.463814 14746 tablet_bootstrap.cc:492] T 67c4b85c17ab4e59be2d6223d993b48c P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:21.463852 14744 raft_consensus.cc:374] T 3d480dd03b6a4d9fb9f64c861d52a720 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:21.463930 14746 ts_tablet_manager.cc:1367] T 67c4b85c17ab4e59be2d6223d993b48c P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 13:56:21.463953 14742 tablet_bootstrap.cc:492] T 3d480dd03b6a4d9fb9f64c861d52a720 P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:21.463986 14744 raft_consensus.cc:729] T 3d480dd03b6a4d9fb9f64c861d52a720 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:21.464020 14742 ts_tablet_manager.cc:1367] T 3d480dd03b6a4d9fb9f64c861d52a720 P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:56:21.464139 14744 consensus_queue.cc:260] T 3d480dd03b6a4d9fb9f64c861d52a720 P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:21.464458 14744 ts_tablet_manager.cc:1398] T 3d480dd03b6a4d9fb9f64c861d52a720 P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:21.464426 14742 raft_consensus.cc:348] T 3d480dd03b6a4d9fb9f64c861d52a720 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:21.464417 14746 raft_consensus.cc:348] T 67c4b85c17ab4e59be2d6223d993b48c P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:21.464540 14742 raft_consensus.cc:374] T 3d480dd03b6a4d9fb9f64c861d52a720 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:21.464540 14746 raft_consensus.cc:374] T 67c4b85c17ab4e59be2d6223d993b48c P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:21.464608 14742 raft_consensus.cc:729] T 3d480dd03b6a4d9fb9f64c861d52a720 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:21.464622 14746 raft_consensus.cc:729] T 67c4b85c17ab4e59be2d6223d993b48c P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:21.464733 14746 consensus_queue.cc:260] T 67c4b85c17ab4e59be2d6223d993b48c P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:21.464745 14742 consensus_queue.cc:260] T 3d480dd03b6a4d9fb9f64c861d52a720 P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:21.465019 14742 ts_tablet_manager.cc:1398] T 3d480dd03b6a4d9fb9f64c861d52a720 P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:21.465094 14746 ts_tablet_manager.cc:1398] T 67c4b85c17ab4e59be2d6223d993b48c P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:21.465209 14742 tablet_bootstrap.cc:492] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:21.465240 14746 tablet_bootstrap.cc:492] T 3d480dd03b6a4d9fb9f64c861d52a720 P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:21.466104 14746 tablet_bootstrap.cc:654] T 3d480dd03b6a4d9fb9f64c861d52a720 P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:21.466104 14742 tablet_bootstrap.cc:654] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:21.468377 14742 tablet_bootstrap.cc:492] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:21.468299 14746 tablet_bootstrap.cc:492] T 3d480dd03b6a4d9fb9f64c861d52a720 P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:21.468463 14742 ts_tablet_manager.cc:1367] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:56:21.468523 14746 ts_tablet_manager.cc:1367] T 3d480dd03b6a4d9fb9f64c861d52a720 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:56:21.468895 14746 raft_consensus.cc:348] T 3d480dd03b6a4d9fb9f64c861d52a720 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:21.468993 14746 raft_consensus.cc:374] T 3d480dd03b6a4d9fb9f64c861d52a720 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:21.469015 14746 raft_consensus.cc:729] T 3d480dd03b6a4d9fb9f64c861d52a720 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:21.469058 14742 raft_consensus.cc:348] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:21.469133 14742 raft_consensus.cc:374] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:21.469153 14742 raft_consensus.cc:729] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:21.469131 14746 consensus_queue.cc:260] T 3d480dd03b6a4d9fb9f64c861d52a720 P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:21.469247 14742 consensus_queue.cc:260] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:21.469388 14746 ts_tablet_manager.cc:1398] T 3d480dd03b6a4d9fb9f64c861d52a720 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:21.469528 14742 ts_tablet_manager.cc:1398] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:21.649773 14680 raft_consensus.cc:482] T 3d480dd03b6a4d9fb9f64c861d52a720 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:21.649910 14680 raft_consensus.cc:504] T 3d480dd03b6a4d9fb9f64c861d52a720 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:21.650352 14680 leader_election.cc:288] T 3d480dd03b6a4d9fb9f64c861d52a720 P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695), ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:21.650750 13053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d480dd03b6a4d9fb9f64c861d52a720" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e" is_pre_election: true
I20240503 13:56:21.650776 13315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d480dd03b6a4d9fb9f64c861d52a720" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703" is_pre_election: true
I20240503 13:56:21.650908 13053 raft_consensus.cc:2454] T 3d480dd03b6a4d9fb9f64c861d52a720 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 0.
I20240503 13:56:21.650931 13315 raft_consensus.cc:2454] T 3d480dd03b6a4d9fb9f64c861d52a720 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 0.
I20240503 13:56:21.651198 13381 leader_election.cc:302] T 3d480dd03b6a4d9fb9f64c861d52a720 P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 7690ee79577c45e3a2c8f8b552c0bff8; no voters:
I20240503 13:56:21.651358 14680 raft_consensus.cc:2781] T 3d480dd03b6a4d9fb9f64c861d52a720 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:21.651422 14680 raft_consensus.cc:482] T 3d480dd03b6a4d9fb9f64c861d52a720 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:21.651468 14680 raft_consensus.cc:3037] T 3d480dd03b6a4d9fb9f64c861d52a720 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:21.652310 14680 raft_consensus.cc:504] T 3d480dd03b6a4d9fb9f64c861d52a720 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:21.652667 14680 leader_election.cc:288] T 3d480dd03b6a4d9fb9f64c861d52a720 P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 election: Requested vote from peers 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695), ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:21.652967 13053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d480dd03b6a4d9fb9f64c861d52a720" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e"
I20240503 13:56:21.653024 13315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d480dd03b6a4d9fb9f64c861d52a720" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703"
I20240503 13:56:21.653074 13053 raft_consensus.cc:3037] T 3d480dd03b6a4d9fb9f64c861d52a720 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:21.653149 13315 raft_consensus.cc:3037] T 3d480dd03b6a4d9fb9f64c861d52a720 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:21.653838 13053 raft_consensus.cc:2454] T 3d480dd03b6a4d9fb9f64c861d52a720 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 1.
I20240503 13:56:21.653904 13315 raft_consensus.cc:2454] T 3d480dd03b6a4d9fb9f64c861d52a720 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 1.
I20240503 13:56:21.654153 13381 leader_election.cc:302] T 3d480dd03b6a4d9fb9f64c861d52a720 P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 7690ee79577c45e3a2c8f8b552c0bff8; no voters:
I20240503 13:56:21.654310 14680 raft_consensus.cc:2781] T 3d480dd03b6a4d9fb9f64c861d52a720 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:21.654397 14680 raft_consensus.cc:686] T 3d480dd03b6a4d9fb9f64c861d52a720 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 LEADER]: Becoming Leader. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Running, Role: LEADER
I20240503 13:56:21.654712 14680 consensus_queue.cc:237] T 3d480dd03b6a4d9fb9f64c861d52a720 P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:21.656466 12903 catalog_manager.cc:5484] T 3d480dd03b6a4d9fb9f64c861d52a720 P 7690ee79577c45e3a2c8f8b552c0bff8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68). New cstate: current_term: 1 leader_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:21.673318 14680 raft_consensus.cc:482] T 67c4b85c17ab4e59be2d6223d993b48c P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:21.673451 14680 raft_consensus.cc:504] T 67c4b85c17ab4e59be2d6223d993b48c P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:21.673867 14680 leader_election.cc:288] T 67c4b85c17ab4e59be2d6223d993b48c P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:21.674304 13184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c4b85c17ab4e59be2d6223d993b48c" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770" is_pre_election: true
I20240503 13:56:21.674285 13053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c4b85c17ab4e59be2d6223d993b48c" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e" is_pre_election: true
I20240503 13:56:21.674435 13053 raft_consensus.cc:2454] T 67c4b85c17ab4e59be2d6223d993b48c P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 0.
I20240503 13:56:21.674435 13184 raft_consensus.cc:2454] T 67c4b85c17ab4e59be2d6223d993b48c P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 0.
I20240503 13:56:21.674708 13381 leader_election.cc:302] T 67c4b85c17ab4e59be2d6223d993b48c P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 7690ee79577c45e3a2c8f8b552c0bff8; no voters:
I20240503 13:56:21.674898 14680 raft_consensus.cc:2781] T 67c4b85c17ab4e59be2d6223d993b48c P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:21.674962 14680 raft_consensus.cc:482] T 67c4b85c17ab4e59be2d6223d993b48c P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:21.674988 14680 raft_consensus.cc:3037] T 67c4b85c17ab4e59be2d6223d993b48c P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:21.675875 14680 raft_consensus.cc:504] T 67c4b85c17ab4e59be2d6223d993b48c P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:21.676204 14680 leader_election.cc:288] T 67c4b85c17ab4e59be2d6223d993b48c P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 election: Requested vote from peers 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:21.676581 13053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c4b85c17ab4e59be2d6223d993b48c" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e"
I20240503 13:56:21.676640 13184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c4b85c17ab4e59be2d6223d993b48c" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770"
I20240503 13:56:21.676702 13053 raft_consensus.cc:3037] T 67c4b85c17ab4e59be2d6223d993b48c P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:21.676760 13184 raft_consensus.cc:3037] T 67c4b85c17ab4e59be2d6223d993b48c P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:21.677492 13184 raft_consensus.cc:2454] T 67c4b85c17ab4e59be2d6223d993b48c P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 1.
I20240503 13:56:21.677564 13053 raft_consensus.cc:2454] T 67c4b85c17ab4e59be2d6223d993b48c P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 1.
I20240503 13:56:21.677845 13381 leader_election.cc:302] T 67c4b85c17ab4e59be2d6223d993b48c P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 7690ee79577c45e3a2c8f8b552c0bff8; no voters:
I20240503 13:56:21.678082 14680 raft_consensus.cc:2781] T 67c4b85c17ab4e59be2d6223d993b48c P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:21.678174 14680 raft_consensus.cc:686] T 67c4b85c17ab4e59be2d6223d993b48c P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 LEADER]: Becoming Leader. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Running, Role: LEADER
I20240503 13:56:21.678489 14680 consensus_queue.cc:237] T 67c4b85c17ab4e59be2d6223d993b48c P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:21.680297 12903 catalog_manager.cc:5484] T 67c4b85c17ab4e59be2d6223d993b48c P 7690ee79577c45e3a2c8f8b552c0bff8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68). New cstate: current_term: 1 leader_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } }
I20240503 13:56:21.794103 14572 raft_consensus.cc:482] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:21.794239 14572 raft_consensus.cc:504] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:21.794611 14572 leader_election.cc:288] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:21.795022 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed3d2ea6e2d74bc5ba60dbbbf2e7b41d" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
I20240503 13:56:21.795034 13053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed3d2ea6e2d74bc5ba60dbbbf2e7b41d" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e" is_pre_election: true
I20240503 13:56:21.795148 13053 raft_consensus.cc:2454] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 0.
I20240503 13:56:21.795159 13446 raft_consensus.cc:2454] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 0.
I20240503 13:56:21.795434 13119 leader_election.cc:302] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 8ccf5a4ac450414c9d3579bd5800b770; no voters:
I20240503 13:56:21.795607 14572 raft_consensus.cc:2781] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:21.795661 14572 raft_consensus.cc:482] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:21.795686 14572 raft_consensus.cc:3037] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:21.796423 14572 raft_consensus.cc:504] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:21.796779 14572 leader_election.cc:288] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 election: Requested vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:21.797132 13053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed3d2ea6e2d74bc5ba60dbbbf2e7b41d" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e"
I20240503 13:56:21.797132 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed3d2ea6e2d74bc5ba60dbbbf2e7b41d" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
I20240503 13:56:21.797256 13446 raft_consensus.cc:3037] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:21.797256 13053 raft_consensus.cc:3037] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:21.798207 13053 raft_consensus.cc:2454] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 1.
I20240503 13:56:21.798207 13446 raft_consensus.cc:2454] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 1.
I20240503 13:56:21.798511 13118 leader_election.cc:302] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, 8ccf5a4ac450414c9d3579bd5800b770; no voters:
I20240503 13:56:21.798683 14572 raft_consensus.cc:2781] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:21.798771 14572 raft_consensus.cc:686] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 LEADER]: Becoming Leader. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Running, Role: LEADER
I20240503 13:56:21.799111 14572 consensus_queue.cc:237] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:21.800978 12903 catalog_manager.cc:5484] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 8ccf5a4ac450414c9d3579bd5800b770 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66). New cstate: current_term: 1 leader_uuid: "8ccf5a4ac450414c9d3579bd5800b770" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } }
W20240503 13:56:21.936250 14758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:56:21.936605 14758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:56:21.939756 14758 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_10'
I20240503 13:56:22.028621 13184 raft_consensus.cc:1261] T 67c4b85c17ab4e59be2d6223d993b48c P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Refusing update from remote peer 7690ee79577c45e3a2c8f8b552c0bff8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:22.028653 13052 raft_consensus.cc:1261] T 3d480dd03b6a4d9fb9f64c861d52a720 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Refusing update from remote peer 7690ee79577c45e3a2c8f8b552c0bff8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:22.028709 13315 raft_consensus.cc:1261] T 3d480dd03b6a4d9fb9f64c861d52a720 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Refusing update from remote peer 7690ee79577c45e3a2c8f8b552c0bff8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:22.029103 14597 consensus_queue.cc:1035] T 67c4b85c17ab4e59be2d6223d993b48c P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:22.029369 14680 consensus_queue.cc:1035] T 3d480dd03b6a4d9fb9f64c861d52a720 P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:22.029470 13053 raft_consensus.cc:1261] T 67c4b85c17ab4e59be2d6223d993b48c P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Refusing update from remote peer 7690ee79577c45e3a2c8f8b552c0bff8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:22.029945 14597 consensus_queue.cc:1035] T 3d480dd03b6a4d9fb9f64c861d52a720 P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:22.030217 14597 consensus_queue.cc:1035] T 67c4b85c17ab4e59be2d6223d993b48c P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:22.033313 13054 raft_consensus.cc:1261] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Refusing update from remote peer 8ccf5a4ac450414c9d3579bd5800b770: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:22.033728 14572 consensus_queue.cc:1035] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:22.037010 13446 raft_consensus.cc:1261] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer 8ccf5a4ac450414c9d3579bd5800b770: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:22.037535 14572 consensus_queue.cc:1035] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:22.069130 14792 mvcc.cc:205] Tried to move back new op lower bound from 7023593807983448064 to 7023593806556397568. Current Snapshot: MvccSnapshot[applied={T|T < 7023593807983448064}]
INSERT report
rows total: 2000
time total: 113.266 ms
time per row: 0.0566331 ms
I20240503 13:56:22.097740 12906 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalance_test_table_11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:56:22.103624 13033 tablet_service.cc:1467] Processing CreateTablet for tablet 66121c347e6c443dac7c5123826a7ae1 (DEFAULT_TABLE table=rebalance_test_table_11 [id=ba5c23cc430441c0a236a4359195fcb0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:22.103960 13033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66121c347e6c443dac7c5123826a7ae1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:22.104166 13032 tablet_service.cc:1467] Processing CreateTablet for tablet 05917bf47aea42d397ccae57c3f29416 (DEFAULT_TABLE table=rebalance_test_table_11 [id=ba5c23cc430441c0a236a4359195fcb0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:22.104429 13032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05917bf47aea42d397ccae57c3f29416. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:22.104701 13295 tablet_service.cc:1467] Processing CreateTablet for tablet 66121c347e6c443dac7c5123826a7ae1 (DEFAULT_TABLE table=rebalance_test_table_11 [id=ba5c23cc430441c0a236a4359195fcb0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:22.104751 13294 tablet_service.cc:1467] Processing CreateTablet for tablet 05917bf47aea42d397ccae57c3f29416 (DEFAULT_TABLE table=rebalance_test_table_11 [id=ba5c23cc430441c0a236a4359195fcb0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:22.104981 13295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66121c347e6c443dac7c5123826a7ae1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:22.105026 13165 tablet_service.cc:1467] Processing CreateTablet for tablet 66121c347e6c443dac7c5123826a7ae1 (DEFAULT_TABLE table=rebalance_test_table_11 [id=ba5c23cc430441c0a236a4359195fcb0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:22.105218 13294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05917bf47aea42d397ccae57c3f29416. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:22.105297 13165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66121c347e6c443dac7c5123826a7ae1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:22.107474 13164 tablet_service.cc:1467] Processing CreateTablet for tablet a746bcb2d718433f8c9acc8bd027a429 (DEFAULT_TABLE table=rebalance_test_table_11 [id=ba5c23cc430441c0a236a4359195fcb0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:22.107692 13164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a746bcb2d718433f8c9acc8bd027a429. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:22.104751 13296 tablet_service.cc:1467] Processing CreateTablet for tablet a746bcb2d718433f8c9acc8bd027a429 (DEFAULT_TABLE table=rebalance_test_table_11 [id=ba5c23cc430441c0a236a4359195fcb0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:22.109663 13296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a746bcb2d718433f8c9acc8bd027a429. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:22.110353 14821 tablet_bootstrap.cc:492] T 05917bf47aea42d397ccae57c3f29416 P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:22.111209 14821 tablet_bootstrap.cc:654] T 05917bf47aea42d397ccae57c3f29416 P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:22.111842 14822 tablet_bootstrap.cc:492] T 66121c347e6c443dac7c5123826a7ae1 P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:22.112813 14822 tablet_bootstrap.cc:654] T 66121c347e6c443dac7c5123826a7ae1 P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:22.112756 13425 tablet_service.cc:1467] Processing CreateTablet for tablet 05917bf47aea42d397ccae57c3f29416 (DEFAULT_TABLE table=rebalance_test_table_11 [id=ba5c23cc430441c0a236a4359195fcb0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:22.113006 13425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05917bf47aea42d397ccae57c3f29416. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:22.113934 14820 tablet_bootstrap.cc:492] T 66121c347e6c443dac7c5123826a7ae1 P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:22.114277 14821 tablet_bootstrap.cc:492] T 05917bf47aea42d397ccae57c3f29416 P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:22.114362 14821 ts_tablet_manager.cc:1367] T 05917bf47aea42d397ccae57c3f29416 P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 13:56:22.114766 14821 raft_consensus.cc:348] T 05917bf47aea42d397ccae57c3f29416 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:22.114871 14821 raft_consensus.cc:374] T 05917bf47aea42d397ccae57c3f29416 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:22.114912 14821 raft_consensus.cc:729] T 05917bf47aea42d397ccae57c3f29416 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:22.115062 14821 consensus_queue.cc:260] T 05917bf47aea42d397ccae57c3f29416 P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:22.115192 14820 tablet_bootstrap.cc:654] T 66121c347e6c443dac7c5123826a7ae1 P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:22.115252 13426 tablet_service.cc:1467] Processing CreateTablet for tablet a746bcb2d718433f8c9acc8bd027a429 (DEFAULT_TABLE table=rebalance_test_table_11 [id=ba5c23cc430441c0a236a4359195fcb0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:22.115478 13426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a746bcb2d718433f8c9acc8bd027a429. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:22.115550 14822 tablet_bootstrap.cc:492] T 66121c347e6c443dac7c5123826a7ae1 P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:22.115633 14822 ts_tablet_manager.cc:1367] T 66121c347e6c443dac7c5123826a7ae1 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:56:22.116076 14822 raft_consensus.cc:348] T 66121c347e6c443dac7c5123826a7ae1 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:22.116181 14822 raft_consensus.cc:374] T 66121c347e6c443dac7c5123826a7ae1 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:22.116211 14822 raft_consensus.cc:729] T 66121c347e6c443dac7c5123826a7ae1 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:22.116240 14820 tablet_bootstrap.cc:492] T 66121c347e6c443dac7c5123826a7ae1 P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:22.116315 14820 ts_tablet_manager.cc:1367] T 66121c347e6c443dac7c5123826a7ae1 P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:56:22.116609 14821 ts_tablet_manager.cc:1398] T 05917bf47aea42d397ccae57c3f29416 P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:22.116797 14821 tablet_bootstrap.cc:492] T 66121c347e6c443dac7c5123826a7ae1 P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:22.116799 14820 raft_consensus.cc:348] T 66121c347e6c443dac7c5123826a7ae1 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:22.116892 14820 raft_consensus.cc:374] T 66121c347e6c443dac7c5123826a7ae1 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:22.116921 14820 raft_consensus.cc:729] T 66121c347e6c443dac7c5123826a7ae1 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:22.117041 14820 consensus_queue.cc:260] T 66121c347e6c443dac7c5123826a7ae1 P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:22.117296 14820 ts_tablet_manager.cc:1398] T 66121c347e6c443dac7c5123826a7ae1 P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:22.117465 14820 tablet_bootstrap.cc:492] T 05917bf47aea42d397ccae57c3f29416 P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:22.118189 14825 tablet_bootstrap.cc:492] T 05917bf47aea42d397ccae57c3f29416 P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:22.118451 14820 tablet_bootstrap.cc:654] T 05917bf47aea42d397ccae57c3f29416 P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:22.118705 14759 raft_consensus.cc:482] T 05917bf47aea42d397ccae57c3f29416 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:22.119072 14825 tablet_bootstrap.cc:654] T 05917bf47aea42d397ccae57c3f29416 P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:22.118912 14759 raft_consensus.cc:504] T 05917bf47aea42d397ccae57c3f29416 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:22.119505 14759 leader_election.cc:288] T 05917bf47aea42d397ccae57c3f29416 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:22.119841 13054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05917bf47aea42d397ccae57c3f29416" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e" is_pre_election: true
I20240503 13:56:22.120150 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05917bf47aea42d397ccae57c3f29416" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
W20240503 13:56:22.120733 13249 leader_election.cc:341] T 05917bf47aea42d397ccae57c3f29416 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109): Illegal state: must be running to vote when last-logged opid is not known
W20240503 13:56:22.120877 13250 leader_election.cc:341] T 05917bf47aea42d397ccae57c3f29416 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695): Illegal state: must be running to vote when last-logged opid is not known
I20240503 13:56:22.120929 13250 leader_election.cc:302] T 05917bf47aea42d397ccae57c3f29416 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba0194098aa84449ae17e8a0d4c43703; no voters: 60ba3ffe3b7746469834cd91c84be14e, 7690ee79577c45e3a2c8f8b552c0bff8
I20240503 13:56:22.121127 14759 raft_consensus.cc:2726] T 05917bf47aea42d397ccae57c3f29416 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 13:56:22.121934 14822 consensus_queue.cc:260] T 66121c347e6c443dac7c5123826a7ae1 P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:22.124917 14821 tablet_bootstrap.cc:654] T 66121c347e6c443dac7c5123826a7ae1 P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:22.125831 14822 ts_tablet_manager.cc:1398] T 66121c347e6c443dac7c5123826a7ae1 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20240503 13:56:22.126096 14822 tablet_bootstrap.cc:492] T a746bcb2d718433f8c9acc8bd027a429 P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:22.126169 14821 tablet_bootstrap.cc:492] T 66121c347e6c443dac7c5123826a7ae1 P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:22.126282 14821 ts_tablet_manager.cc:1367] T 66121c347e6c443dac7c5123826a7ae1 P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 13:56:22.126750 14821 raft_consensus.cc:348] T 66121c347e6c443dac7c5123826a7ae1 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:22.126883 14821 raft_consensus.cc:374] T 66121c347e6c443dac7c5123826a7ae1 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:22.126955 14821 raft_consensus.cc:729] T 66121c347e6c443dac7c5123826a7ae1 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:22.127101 14821 consensus_queue.cc:260] T 66121c347e6c443dac7c5123826a7ae1 P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:22.128031 14821 ts_tablet_manager.cc:1398] T 66121c347e6c443dac7c5123826a7ae1 P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:22.128257 14821 tablet_bootstrap.cc:492] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:22.129259 14821 tablet_bootstrap.cc:654] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:22.130033 14822 tablet_bootstrap.cc:654] T a746bcb2d718433f8c9acc8bd027a429 P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:22.130064 14825 tablet_bootstrap.cc:492] T 05917bf47aea42d397ccae57c3f29416 P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:22.130263 14825 ts_tablet_manager.cc:1367] T 05917bf47aea42d397ccae57c3f29416 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 13:56:22.130895 14825 raft_consensus.cc:348] T 05917bf47aea42d397ccae57c3f29416 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:22.131057 14821 tablet_bootstrap.cc:492] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:22.131137 14821 ts_tablet_manager.cc:1367] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:56:22.131320 14820 tablet_bootstrap.cc:492] T 05917bf47aea42d397ccae57c3f29416 P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:22.131417 14820 ts_tablet_manager.cc:1367] T 05917bf47aea42d397ccae57c3f29416 P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240503 13:56:22.131541 14825 raft_consensus.cc:374] T 05917bf47aea42d397ccae57c3f29416 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:22.131595 14825 raft_consensus.cc:729] T 05917bf47aea42d397ccae57c3f29416 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:22.131541 14821 raft_consensus.cc:348] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:22.131642 14821 raft_consensus.cc:374] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:22.131667 14821 raft_consensus.cc:729] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:22.131781 14821 consensus_queue.cc:260] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:22.131841 14820 raft_consensus.cc:348] T 05917bf47aea42d397ccae57c3f29416 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:22.131932 14820 raft_consensus.cc:374] T 05917bf47aea42d397ccae57c3f29416 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:22.131958 14820 raft_consensus.cc:729] T 05917bf47aea42d397ccae57c3f29416 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:22.132002 14825 consensus_queue.cc:260] T 05917bf47aea42d397ccae57c3f29416 P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:22.132066 14821 ts_tablet_manager.cc:1398] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:22.132082 14820 consensus_queue.cc:260] T 05917bf47aea42d397ccae57c3f29416 P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:22.132344 14820 ts_tablet_manager.cc:1398] T 05917bf47aea42d397ccae57c3f29416 P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:22.132362 14825 ts_tablet_manager.cc:1398] T 05917bf47aea42d397ccae57c3f29416 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:22.132602 14825 tablet_bootstrap.cc:492] T a746bcb2d718433f8c9acc8bd027a429 P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:22.133625 14822 tablet_bootstrap.cc:492] T a746bcb2d718433f8c9acc8bd027a429 P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:22.133745 14822 ts_tablet_manager.cc:1367] T a746bcb2d718433f8c9acc8bd027a429 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 13:56:22.134203 14822 raft_consensus.cc:348] T a746bcb2d718433f8c9acc8bd027a429 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:22.134347 14822 raft_consensus.cc:374] T a746bcb2d718433f8c9acc8bd027a429 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:22.134413 14822 raft_consensus.cc:729] T a746bcb2d718433f8c9acc8bd027a429 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:22.134589 14822 consensus_queue.cc:260] T a746bcb2d718433f8c9acc8bd027a429 P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:22.134897 14822 ts_tablet_manager.cc:1398] T a746bcb2d718433f8c9acc8bd027a429 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:22.135738 14825 tablet_bootstrap.cc:654] T a746bcb2d718433f8c9acc8bd027a429 P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:22.136142 14483 raft_consensus.cc:482] T 05917bf47aea42d397ccae57c3f29416 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:22.136235 14483 raft_consensus.cc:504] T 05917bf47aea42d397ccae57c3f29416 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:22.136662 14483 leader_election.cc:288] T 05917bf47aea42d397ccae57c3f29416 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279), 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:22.136940 13314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05917bf47aea42d397ccae57c3f29416" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703" is_pre_election: true
I20240503 13:56:22.136981 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05917bf47aea42d397ccae57c3f29416" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
I20240503 13:56:22.137085 13314 raft_consensus.cc:2454] T 05917bf47aea42d397ccae57c3f29416 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 0.
I20240503 13:56:22.137116 13446 raft_consensus.cc:2454] T 05917bf47aea42d397ccae57c3f29416 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 0.
I20240503 13:56:22.137389 12990 leader_election.cc:302] T 05917bf47aea42d397ccae57c3f29416 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:22.137742 14483 raft_consensus.cc:2781] T 05917bf47aea42d397ccae57c3f29416 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:22.137795 14483 raft_consensus.cc:482] T 05917bf47aea42d397ccae57c3f29416 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:22.137828 14483 raft_consensus.cc:3037] T 05917bf47aea42d397ccae57c3f29416 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:22.137990 14825 tablet_bootstrap.cc:492] T a746bcb2d718433f8c9acc8bd027a429 P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:22.138062 14825 ts_tablet_manager.cc:1367] T a746bcb2d718433f8c9acc8bd027a429 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240503 13:56:22.138386 14825 raft_consensus.cc:348] T a746bcb2d718433f8c9acc8bd027a429 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:22.138464 14825 raft_consensus.cc:374] T a746bcb2d718433f8c9acc8bd027a429 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:22.138494 14825 raft_consensus.cc:729] T a746bcb2d718433f8c9acc8bd027a429 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:22.138588 14825 consensus_queue.cc:260] T a746bcb2d718433f8c9acc8bd027a429 P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:22.138645 14483 raft_consensus.cc:504] T 05917bf47aea42d397ccae57c3f29416 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:22.138806 14825 ts_tablet_manager.cc:1398] T a746bcb2d718433f8c9acc8bd027a429 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:22.138954 14483 leader_election.cc:288] T 05917bf47aea42d397ccae57c3f29416 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 election: Requested vote from peers ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279), 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:22.139241 13314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05917bf47aea42d397ccae57c3f29416" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703"
I20240503 13:56:22.139261 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05917bf47aea42d397ccae57c3f29416" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
I20240503 13:56:22.139354 13446 raft_consensus.cc:3037] T 05917bf47aea42d397ccae57c3f29416 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:22.139354 13314 raft_consensus.cc:3037] T 05917bf47aea42d397ccae57c3f29416 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:22.140229 13446 raft_consensus.cc:2454] T 05917bf47aea42d397ccae57c3f29416 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 1.
I20240503 13:56:22.140229 13314 raft_consensus.cc:2454] T 05917bf47aea42d397ccae57c3f29416 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 1.
I20240503 13:56:22.140549 12987 leader_election.cc:302] T 05917bf47aea42d397ccae57c3f29416 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 7690ee79577c45e3a2c8f8b552c0bff8; no voters:
I20240503 13:56:22.140686 14483 raft_consensus.cc:2781] T 05917bf47aea42d397ccae57c3f29416 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:22.140862 14483 raft_consensus.cc:686] T 05917bf47aea42d397ccae57c3f29416 P 60ba3ffe3b7746469834cd91c84be14e [term 1 LEADER]: Becoming Leader. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Running, Role: LEADER
I20240503 13:56:22.141014 14483 consensus_queue.cc:237] T 05917bf47aea42d397ccae57c3f29416 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:22.142747 12903 catalog_manager.cc:5484] T 05917bf47aea42d397ccae57c3f29416 P 60ba3ffe3b7746469834cd91c84be14e reported cstate change: term changed from 0 to 1, leader changed from <none> to 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65). New cstate: current_term: 1 leader_uuid: "60ba3ffe3b7746469834cd91c84be14e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } }
I20240503 13:56:22.187172 14759 raft_consensus.cc:482] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:22.187302 14759 raft_consensus.cc:504] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:22.187773 14759 leader_election.cc:288] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:22.188185 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a746bcb2d718433f8c9acc8bd027a429" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
I20240503 13:56:22.188215 13184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a746bcb2d718433f8c9acc8bd027a429" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770" is_pre_election: true
I20240503 13:56:22.188333 13184 raft_consensus.cc:2454] T a746bcb2d718433f8c9acc8bd027a429 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 0.
I20240503 13:56:22.188339 13446 raft_consensus.cc:2454] T a746bcb2d718433f8c9acc8bd027a429 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 0.
I20240503 13:56:22.188660 13249 leader_election.cc:302] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:22.188836 14759 raft_consensus.cc:2781] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:22.188905 14759 raft_consensus.cc:482] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:22.188951 14759 raft_consensus.cc:3037] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:22.189764 14759 raft_consensus.cc:504] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:22.190119 14759 leader_election.cc:288] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: Requested vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:22.190400 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a746bcb2d718433f8c9acc8bd027a429" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
I20240503 13:56:22.190443 13184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a746bcb2d718433f8c9acc8bd027a429" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770"
I20240503 13:56:22.190515 13446 raft_consensus.cc:3037] T a746bcb2d718433f8c9acc8bd027a429 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:22.190538 13184 raft_consensus.cc:3037] T a746bcb2d718433f8c9acc8bd027a429 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:22.191246 13446 raft_consensus.cc:2454] T a746bcb2d718433f8c9acc8bd027a429 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 1.
I20240503 13:56:22.191299 13184 raft_consensus.cc:2454] T a746bcb2d718433f8c9acc8bd027a429 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 1.
I20240503 13:56:22.191519 13249 leader_election.cc:302] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:22.191664 14759 raft_consensus.cc:2781] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:22.191771 14759 raft_consensus.cc:686] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Becoming Leader. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Running, Role: LEADER
I20240503 13:56:22.191919 14759 consensus_queue.cc:237] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:22.193729 12903 catalog_manager.cc:5484] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 reported cstate change: term changed from 0 to 1, leader changed from <none> to ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67). New cstate: current_term: 1 leader_uuid: "ba0194098aa84449ae17e8a0d4c43703" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } }
I20240503 13:56:22.476079 14483 raft_consensus.cc:482] T 66121c347e6c443dac7c5123826a7ae1 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:22.476217 14483 raft_consensus.cc:504] T 66121c347e6c443dac7c5123826a7ae1 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:22.476680 14483 leader_election.cc:288] T 66121c347e6c443dac7c5123826a7ae1 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:22.477166 13314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66121c347e6c443dac7c5123826a7ae1" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703" is_pre_election: true
I20240503 13:56:22.477191 13184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66121c347e6c443dac7c5123826a7ae1" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770" is_pre_election: true
I20240503 13:56:22.477313 13314 raft_consensus.cc:2454] T 66121c347e6c443dac7c5123826a7ae1 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 0.
I20240503 13:56:22.477313 13184 raft_consensus.cc:2454] T 66121c347e6c443dac7c5123826a7ae1 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 0.
I20240503 13:56:22.477592 12990 leader_election.cc:302] T 66121c347e6c443dac7c5123826a7ae1 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 8ccf5a4ac450414c9d3579bd5800b770; no voters:
I20240503 13:56:22.477749 14483 raft_consensus.cc:2781] T 66121c347e6c443dac7c5123826a7ae1 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:22.477818 14483 raft_consensus.cc:482] T 66121c347e6c443dac7c5123826a7ae1 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:22.477856 14483 raft_consensus.cc:3037] T 66121c347e6c443dac7c5123826a7ae1 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:22.479049 14483 raft_consensus.cc:504] T 66121c347e6c443dac7c5123826a7ae1 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:22.479386 14483 leader_election.cc:288] T 66121c347e6c443dac7c5123826a7ae1 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 election: Requested vote from peers ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:22.479773 13314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66121c347e6c443dac7c5123826a7ae1" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703"
I20240503 13:56:22.479797 13184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66121c347e6c443dac7c5123826a7ae1" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770"
I20240503 13:56:22.479914 13184 raft_consensus.cc:3037] T 66121c347e6c443dac7c5123826a7ae1 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:22.479914 13314 raft_consensus.cc:3037] T 66121c347e6c443dac7c5123826a7ae1 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:22.480890 13184 raft_consensus.cc:2454] T 66121c347e6c443dac7c5123826a7ae1 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 1.
I20240503 13:56:22.480890 13314 raft_consensus.cc:2454] T 66121c347e6c443dac7c5123826a7ae1 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 1.
I20240503 13:56:22.481186 12990 leader_election.cc:302] T 66121c347e6c443dac7c5123826a7ae1 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 8ccf5a4ac450414c9d3579bd5800b770; no voters:
I20240503 13:56:22.481329 14483 raft_consensus.cc:2781] T 66121c347e6c443dac7c5123826a7ae1 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:22.481415 14483 raft_consensus.cc:686] T 66121c347e6c443dac7c5123826a7ae1 P 60ba3ffe3b7746469834cd91c84be14e [term 1 LEADER]: Becoming Leader. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Running, Role: LEADER
I20240503 13:56:22.481559 14483 consensus_queue.cc:237] T 66121c347e6c443dac7c5123826a7ae1 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:22.483306 12903 catalog_manager.cc:5484] T 66121c347e6c443dac7c5123826a7ae1 P 60ba3ffe3b7746469834cd91c84be14e reported cstate change: term changed from 0 to 1, leader changed from <none> to 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65). New cstate: current_term: 1 leader_uuid: "60ba3ffe3b7746469834cd91c84be14e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } }
I20240503 13:56:22.572116 14483 consensus_queue.cc:1035] T 05917bf47aea42d397ccae57c3f29416 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:22.576140 14483 consensus_queue.cc:1035] T 05917bf47aea42d397ccae57c3f29416 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:22.579344 14759 consensus_queue.cc:1035] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:22.589897 14759 consensus_queue.cc:1035] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 13:56:22.710095 14837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:56:22.710322 14837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:56:22.713445 14837 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_11'
I20240503 13:56:22.821784 13315 raft_consensus.cc:1261] T 66121c347e6c443dac7c5123826a7ae1 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Refusing update from remote peer 60ba3ffe3b7746469834cd91c84be14e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:56:22.822418 13184 raft_consensus.cc:1261] T 66121c347e6c443dac7c5123826a7ae1 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Refusing update from remote peer 60ba3ffe3b7746469834cd91c84be14e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:22.822966 14876 consensus_queue.cc:1035] T 66121c347e6c443dac7c5123826a7ae1 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:22.825513 14876 consensus_queue.cc:1035] T 66121c347e6c443dac7c5123826a7ae1 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 119.151 ms
time per row: 0.0595754 ms
I20240503 13:56:22.880465 12906 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalance_test_table_12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:56:22.886554 13164 tablet_service.cc:1467] Processing CreateTablet for tablet 97dd13a5495a4ec7818d136e7cc95abd (DEFAULT_TABLE table=rebalance_test_table_12 [id=3b862f5ca43e4df4815648fcc0bf383a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:22.886597 13032 tablet_service.cc:1467] Processing CreateTablet for tablet 97dd13a5495a4ec7818d136e7cc95abd (DEFAULT_TABLE table=rebalance_test_table_12 [id=3b862f5ca43e4df4815648fcc0bf383a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:22.886930 13164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97dd13a5495a4ec7818d136e7cc95abd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:22.886945 13032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97dd13a5495a4ec7818d136e7cc95abd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:22.887416 13033 tablet_service.cc:1467] Processing CreateTablet for tablet 2a7bc6049234402d806ce06fdb8df987 (DEFAULT_TABLE table=rebalance_test_table_12 [id=3b862f5ca43e4df4815648fcc0bf383a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:22.887526 13034 tablet_service.cc:1467] Processing CreateTablet for tablet 7a1c36d4fd0049139c98db7af595bc62 (DEFAULT_TABLE table=rebalance_test_table_12 [id=3b862f5ca43e4df4815648fcc0bf383a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:22.887638 13033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a7bc6049234402d806ce06fdb8df987. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:22.887758 13034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a1c36d4fd0049139c98db7af595bc62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:22.889861 13426 tablet_service.cc:1467] Processing CreateTablet for tablet 2a7bc6049234402d806ce06fdb8df987 (DEFAULT_TABLE table=rebalance_test_table_12 [id=3b862f5ca43e4df4815648fcc0bf383a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:22.890179 13426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a7bc6049234402d806ce06fdb8df987. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:22.890246 13296 tablet_service.cc:1467] Processing CreateTablet for tablet 97dd13a5495a4ec7818d136e7cc95abd (DEFAULT_TABLE table=rebalance_test_table_12 [id=3b862f5ca43e4df4815648fcc0bf383a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:22.890515 13296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97dd13a5495a4ec7818d136e7cc95abd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:22.890544 14896 tablet_bootstrap.cc:492] T 97dd13a5495a4ec7818d136e7cc95abd P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:22.890774 14897 tablet_bootstrap.cc:492] T 97dd13a5495a4ec7818d136e7cc95abd P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:22.891659 14897 tablet_bootstrap.cc:654] T 97dd13a5495a4ec7818d136e7cc95abd P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:22.892612 13295 tablet_service.cc:1467] Processing CreateTablet for tablet 7a1c36d4fd0049139c98db7af595bc62 (DEFAULT_TABLE table=rebalance_test_table_12 [id=3b862f5ca43e4df4815648fcc0bf383a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:22.892724 13425 tablet_service.cc:1467] Processing CreateTablet for tablet 7a1c36d4fd0049139c98db7af595bc62 (DEFAULT_TABLE table=rebalance_test_table_12 [id=3b862f5ca43e4df4815648fcc0bf383a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:22.892840 13295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a1c36d4fd0049139c98db7af595bc62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:22.892957 13425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a1c36d4fd0049139c98db7af595bc62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:22.892992 14896 tablet_bootstrap.cc:654] T 97dd13a5495a4ec7818d136e7cc95abd P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:22.893905 14897 tablet_bootstrap.cc:492] T 97dd13a5495a4ec7818d136e7cc95abd P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:22.893993 14897 ts_tablet_manager.cc:1367] T 97dd13a5495a4ec7818d136e7cc95abd P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:56:22.894469 14897 raft_consensus.cc:348] T 97dd13a5495a4ec7818d136e7cc95abd P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:22.894613 14897 raft_consensus.cc:374] T 97dd13a5495a4ec7818d136e7cc95abd P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:22.895048 14896 tablet_bootstrap.cc:492] T 97dd13a5495a4ec7818d136e7cc95abd P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:22.895186 14896 ts_tablet_manager.cc:1367] T 97dd13a5495a4ec7818d136e7cc95abd P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:56:22.895493 14900 tablet_bootstrap.cc:492] T 2a7bc6049234402d806ce06fdb8df987 P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:22.895682 14896 raft_consensus.cc:348] T 97dd13a5495a4ec7818d136e7cc95abd P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:22.895243 13164 tablet_service.cc:1467] Processing CreateTablet for tablet 2a7bc6049234402d806ce06fdb8df987 (DEFAULT_TABLE table=rebalance_test_table_12 [id=3b862f5ca43e4df4815648fcc0bf383a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:22.896006 13164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a7bc6049234402d806ce06fdb8df987. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:22.896387 14900 tablet_bootstrap.cc:654] T 2a7bc6049234402d806ce06fdb8df987 P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:22.896167 14899 tablet_bootstrap.cc:492] T 97dd13a5495a4ec7818d136e7cc95abd P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:22.897513 14899 tablet_bootstrap.cc:654] T 97dd13a5495a4ec7818d136e7cc95abd P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:22.897598 14900 tablet_bootstrap.cc:492] T 2a7bc6049234402d806ce06fdb8df987 P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:22.897693 14900 ts_tablet_manager.cc:1367] T 2a7bc6049234402d806ce06fdb8df987 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:22.898072 14900 raft_consensus.cc:348] T 2a7bc6049234402d806ce06fdb8df987 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:22.898182 14900 raft_consensus.cc:374] T 2a7bc6049234402d806ce06fdb8df987 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:22.898213 14900 raft_consensus.cc:729] T 2a7bc6049234402d806ce06fdb8df987 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:22.898322 14900 consensus_queue.cc:260] T 2a7bc6049234402d806ce06fdb8df987 P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:22.898574 14896 raft_consensus.cc:374] T 97dd13a5495a4ec7818d136e7cc95abd P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:22.898591 14900 ts_tablet_manager.cc:1398] T 2a7bc6049234402d806ce06fdb8df987 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:22.898628 14896 raft_consensus.cc:729] T 97dd13a5495a4ec7818d136e7cc95abd P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:22.898653 14899 tablet_bootstrap.cc:492] T 97dd13a5495a4ec7818d136e7cc95abd P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:22.898722 14899 ts_tablet_manager.cc:1367] T 97dd13a5495a4ec7818d136e7cc95abd P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:56:22.898759 14900 tablet_bootstrap.cc:492] T 7a1c36d4fd0049139c98db7af595bc62 P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:22.898759 14896 consensus_queue.cc:260] T 97dd13a5495a4ec7818d136e7cc95abd P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:22.899065 14896 ts_tablet_manager.cc:1398] T 97dd13a5495a4ec7818d136e7cc95abd P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 13:56:22.899121 14899 raft_consensus.cc:348] T 97dd13a5495a4ec7818d136e7cc95abd P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:22.899207 14899 raft_consensus.cc:374] T 97dd13a5495a4ec7818d136e7cc95abd P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:22.899232 14896 tablet_bootstrap.cc:492] T 2a7bc6049234402d806ce06fdb8df987 P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:22.899235 14899 raft_consensus.cc:729] T 97dd13a5495a4ec7818d136e7cc95abd P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:22.899360 14899 consensus_queue.cc:260] T 97dd13a5495a4ec7818d136e7cc95abd P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:22.899446 14900 tablet_bootstrap.cc:654] T 7a1c36d4fd0049139c98db7af595bc62 P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:22.899616 14899 ts_tablet_manager.cc:1398] T 97dd13a5495a4ec7818d136e7cc95abd P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:22.899785 14899 tablet_bootstrap.cc:492] T 7a1c36d4fd0049139c98db7af595bc62 P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:22.900084 14896 tablet_bootstrap.cc:654] T 2a7bc6049234402d806ce06fdb8df987 P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:22.901101 14899 tablet_bootstrap.cc:654] T 7a1c36d4fd0049139c98db7af595bc62 P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:22.901856 14900 tablet_bootstrap.cc:492] T 7a1c36d4fd0049139c98db7af595bc62 P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:22.901930 14900 ts_tablet_manager.cc:1367] T 7a1c36d4fd0049139c98db7af595bc62 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:56:22.902060 14897 raft_consensus.cc:729] T 97dd13a5495a4ec7818d136e7cc95abd P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:22.902267 14897 consensus_queue.cc:260] T 97dd13a5495a4ec7818d136e7cc95abd P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:22.902385 14900 raft_consensus.cc:348] T 7a1c36d4fd0049139c98db7af595bc62 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:22.902875 14899 tablet_bootstrap.cc:492] T 7a1c36d4fd0049139c98db7af595bc62 P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:22.902928 14897 ts_tablet_manager.cc:1398] T 97dd13a5495a4ec7818d136e7cc95abd P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20240503 13:56:22.902966 14899 ts_tablet_manager.cc:1367] T 7a1c36d4fd0049139c98db7af595bc62 P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:56:22.903376 14899 raft_consensus.cc:348] T 7a1c36d4fd0049139c98db7af595bc62 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:22.903466 14899 raft_consensus.cc:374] T 7a1c36d4fd0049139c98db7af595bc62 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:22.903496 14899 raft_consensus.cc:729] T 7a1c36d4fd0049139c98db7af595bc62 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:22.902485 14900 raft_consensus.cc:374] T 7a1c36d4fd0049139c98db7af595bc62 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:22.903584 14900 raft_consensus.cc:729] T 7a1c36d4fd0049139c98db7af595bc62 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:22.903617 14899 consensus_queue.cc:260] T 7a1c36d4fd0049139c98db7af595bc62 P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:22.903702 14900 consensus_queue.cc:260] T 7a1c36d4fd0049139c98db7af595bc62 P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:22.903888 14899 ts_tablet_manager.cc:1398] T 7a1c36d4fd0049139c98db7af595bc62 P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:22.903951 14900 ts_tablet_manager.cc:1398] T 7a1c36d4fd0049139c98db7af595bc62 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:22.906455 14897 tablet_bootstrap.cc:492] T 2a7bc6049234402d806ce06fdb8df987 P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:22.907266 14897 tablet_bootstrap.cc:654] T 2a7bc6049234402d806ce06fdb8df987 P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:22.908610 14897 tablet_bootstrap.cc:492] T 2a7bc6049234402d806ce06fdb8df987 P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:22.908700 14897 ts_tablet_manager.cc:1367] T 2a7bc6049234402d806ce06fdb8df987 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:56:22.909099 14897 raft_consensus.cc:348] T 2a7bc6049234402d806ce06fdb8df987 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:22.909209 14897 raft_consensus.cc:374] T 2a7bc6049234402d806ce06fdb8df987 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:22.909237 14897 raft_consensus.cc:729] T 2a7bc6049234402d806ce06fdb8df987 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:22.909670 14896 tablet_bootstrap.cc:492] T 2a7bc6049234402d806ce06fdb8df987 P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:22.909759 14896 ts_tablet_manager.cc:1367] T 2a7bc6049234402d806ce06fdb8df987 P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20240503 13:56:22.910298 14896 raft_consensus.cc:348] T 2a7bc6049234402d806ce06fdb8df987 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:22.910406 14896 raft_consensus.cc:374] T 2a7bc6049234402d806ce06fdb8df987 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:22.910434 14896 raft_consensus.cc:729] T 2a7bc6049234402d806ce06fdb8df987 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:22.910573 14896 consensus_queue.cc:260] T 2a7bc6049234402d806ce06fdb8df987 P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:22.910846 14896 ts_tablet_manager.cc:1398] T 2a7bc6049234402d806ce06fdb8df987 P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:22.911005 14897 consensus_queue.cc:260] T 2a7bc6049234402d806ce06fdb8df987 P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:22.911367 14897 ts_tablet_manager.cc:1398] T 2a7bc6049234402d806ce06fdb8df987 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:56:22.911582 14896 tablet_bootstrap.cc:492] T 7a1c36d4fd0049139c98db7af595bc62 P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:22.912561 14896 tablet_bootstrap.cc:654] T 7a1c36d4fd0049139c98db7af595bc62 P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:22.913707 14896 tablet_bootstrap.cc:492] T 7a1c36d4fd0049139c98db7af595bc62 P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:22.913816 14896 ts_tablet_manager.cc:1367] T 7a1c36d4fd0049139c98db7af595bc62 P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 13:56:22.914158 14896 raft_consensus.cc:348] T 7a1c36d4fd0049139c98db7af595bc62 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:22.914279 14896 raft_consensus.cc:374] T 7a1c36d4fd0049139c98db7af595bc62 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:22.914326 14896 raft_consensus.cc:729] T 7a1c36d4fd0049139c98db7af595bc62 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:22.914472 14896 consensus_queue.cc:260] T 7a1c36d4fd0049139c98db7af595bc62 P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:22.914705 14896 ts_tablet_manager.cc:1398] T 7a1c36d4fd0049139c98db7af595bc62 P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:22.946136 14438 raft_consensus.cc:482] T 97dd13a5495a4ec7818d136e7cc95abd P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:22.946278 14438 raft_consensus.cc:504] T 97dd13a5495a4ec7818d136e7cc95abd P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:22.946774 14438 leader_election.cc:288] T 97dd13a5495a4ec7818d136e7cc95abd P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387), 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:22.947196 13184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97dd13a5495a4ec7818d136e7cc95abd" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770" is_pre_election: true
I20240503 13:56:22.947285 13054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97dd13a5495a4ec7818d136e7cc95abd" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e" is_pre_election: true
I20240503 13:56:22.947355 13184 raft_consensus.cc:2454] T 97dd13a5495a4ec7818d136e7cc95abd P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 0.
I20240503 13:56:22.947412 13054 raft_consensus.cc:2454] T 97dd13a5495a4ec7818d136e7cc95abd P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 0.
I20240503 13:56:22.947690 13250 leader_election.cc:302] T 97dd13a5495a4ec7818d136e7cc95abd P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:22.947870 14438 raft_consensus.cc:2781] T 97dd13a5495a4ec7818d136e7cc95abd P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:22.947933 14438 raft_consensus.cc:482] T 97dd13a5495a4ec7818d136e7cc95abd P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:22.947965 14438 raft_consensus.cc:3037] T 97dd13a5495a4ec7818d136e7cc95abd P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:22.948856 14438 raft_consensus.cc:504] T 97dd13a5495a4ec7818d136e7cc95abd P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:22.949195 14438 leader_election.cc:288] T 97dd13a5495a4ec7818d136e7cc95abd P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: Requested vote from peers 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387), 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:22.949501 13054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97dd13a5495a4ec7818d136e7cc95abd" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e"
I20240503 13:56:22.949524 13184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97dd13a5495a4ec7818d136e7cc95abd" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770"
I20240503 13:56:22.949630 13184 raft_consensus.cc:3037] T 97dd13a5495a4ec7818d136e7cc95abd P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:22.949626 13054 raft_consensus.cc:3037] T 97dd13a5495a4ec7818d136e7cc95abd P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:22.950316 13184 raft_consensus.cc:2454] T 97dd13a5495a4ec7818d136e7cc95abd P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 1.
I20240503 13:56:22.950453 13054 raft_consensus.cc:2454] T 97dd13a5495a4ec7818d136e7cc95abd P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 1.
I20240503 13:56:22.950629 13252 leader_election.cc:302] T 97dd13a5495a4ec7818d136e7cc95abd P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ccf5a4ac450414c9d3579bd5800b770, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:22.950783 14438 raft_consensus.cc:2781] T 97dd13a5495a4ec7818d136e7cc95abd P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:22.950862 14438 raft_consensus.cc:686] T 97dd13a5495a4ec7818d136e7cc95abd P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Becoming Leader. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Running, Role: LEADER
I20240503 13:56:22.951006 14438 consensus_queue.cc:237] T 97dd13a5495a4ec7818d136e7cc95abd P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:22.952795 12912 catalog_manager.cc:5484] T 97dd13a5495a4ec7818d136e7cc95abd P ba0194098aa84449ae17e8a0d4c43703 reported cstate change: term changed from 0 to 1, leader changed from <none> to ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67). New cstate: current_term: 1 leader_uuid: "ba0194098aa84449ae17e8a0d4c43703" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:23.042564 14876 raft_consensus.cc:482] T 7a1c36d4fd0049139c98db7af595bc62 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:23.042730 14876 raft_consensus.cc:504] T 7a1c36d4fd0049139c98db7af595bc62 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:23.043231 14876 leader_election.cc:288] T 7a1c36d4fd0049139c98db7af595bc62 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:23.043560 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a1c36d4fd0049139c98db7af595bc62" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
I20240503 13:56:23.043627 13314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a1c36d4fd0049139c98db7af595bc62" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703" is_pre_election: true
I20240503 13:56:23.043710 13446 raft_consensus.cc:2454] T 7a1c36d4fd0049139c98db7af595bc62 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 0.
I20240503 13:56:23.043776 13314 raft_consensus.cc:2454] T 7a1c36d4fd0049139c98db7af595bc62 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 0.
I20240503 13:56:23.043954 12987 leader_election.cc:302] T 7a1c36d4fd0049139c98db7af595bc62 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 7690ee79577c45e3a2c8f8b552c0bff8; no voters:
I20240503 13:56:23.044131 14876 raft_consensus.cc:2781] T 7a1c36d4fd0049139c98db7af595bc62 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:23.044203 14876 raft_consensus.cc:482] T 7a1c36d4fd0049139c98db7af595bc62 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:23.044236 14876 raft_consensus.cc:3037] T 7a1c36d4fd0049139c98db7af595bc62 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:23.045202 14876 raft_consensus.cc:504] T 7a1c36d4fd0049139c98db7af595bc62 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:23.045550 14876 leader_election.cc:288] T 7a1c36d4fd0049139c98db7af595bc62 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 election: Requested vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:23.045922 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a1c36d4fd0049139c98db7af595bc62" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
I20240503 13:56:23.045914 13314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a1c36d4fd0049139c98db7af595bc62" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703"
I20240503 13:56:23.046030 13446 raft_consensus.cc:3037] T 7a1c36d4fd0049139c98db7af595bc62 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:23.046039 13314 raft_consensus.cc:3037] T 7a1c36d4fd0049139c98db7af595bc62 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:23.046780 13446 raft_consensus.cc:2454] T 7a1c36d4fd0049139c98db7af595bc62 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 1.
I20240503 13:56:23.046782 13314 raft_consensus.cc:2454] T 7a1c36d4fd0049139c98db7af595bc62 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 1.
I20240503 13:56:23.047082 12990 leader_election.cc:302] T 7a1c36d4fd0049139c98db7af595bc62 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:23.047246 14876 raft_consensus.cc:2781] T 7a1c36d4fd0049139c98db7af595bc62 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:23.047343 14876 raft_consensus.cc:686] T 7a1c36d4fd0049139c98db7af595bc62 P 60ba3ffe3b7746469834cd91c84be14e [term 1 LEADER]: Becoming Leader. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Running, Role: LEADER
I20240503 13:56:23.047581 14876 consensus_queue.cc:237] T 7a1c36d4fd0049139c98db7af595bc62 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:23.049628 12912 catalog_manager.cc:5484] T 7a1c36d4fd0049139c98db7af595bc62 P 60ba3ffe3b7746469834cd91c84be14e reported cstate change: term changed from 0 to 1, leader changed from <none> to 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65). New cstate: current_term: 1 leader_uuid: "60ba3ffe3b7746469834cd91c84be14e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:23.235867 14793 raft_consensus.cc:482] T 2a7bc6049234402d806ce06fdb8df987 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:23.236020 14793 raft_consensus.cc:504] T 2a7bc6049234402d806ce06fdb8df987 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:23.236385 14793 leader_election.cc:288] T 2a7bc6049234402d806ce06fdb8df987 P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:23.236757 13184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a7bc6049234402d806ce06fdb8df987" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770" is_pre_election: true
I20240503 13:56:23.236888 13184 raft_consensus.cc:2454] T 2a7bc6049234402d806ce06fdb8df987 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 0.
I20240503 13:56:23.236833 13054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a7bc6049234402d806ce06fdb8df987" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e" is_pre_election: true
I20240503 13:56:23.236997 13054 raft_consensus.cc:2454] T 2a7bc6049234402d806ce06fdb8df987 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 0.
I20240503 13:56:23.237214 13383 leader_election.cc:302] T 2a7bc6049234402d806ce06fdb8df987 P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, 8ccf5a4ac450414c9d3579bd5800b770; no voters:
I20240503 13:56:23.237468 14793 raft_consensus.cc:2781] T 2a7bc6049234402d806ce06fdb8df987 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:23.237529 14793 raft_consensus.cc:482] T 2a7bc6049234402d806ce06fdb8df987 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:23.237571 14793 raft_consensus.cc:3037] T 2a7bc6049234402d806ce06fdb8df987 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:23.238577 14793 raft_consensus.cc:504] T 2a7bc6049234402d806ce06fdb8df987 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:23.238891 14793 leader_election.cc:288] T 2a7bc6049234402d806ce06fdb8df987 P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 election: Requested vote from peers 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:23.239179 13054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a7bc6049234402d806ce06fdb8df987" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e"
I20240503 13:56:23.239280 13054 raft_consensus.cc:3037] T 2a7bc6049234402d806ce06fdb8df987 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:23.240012 13054 raft_consensus.cc:2454] T 2a7bc6049234402d806ce06fdb8df987 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 1.
I20240503 13:56:23.240289 13184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a7bc6049234402d806ce06fdb8df987" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770"
I20240503 13:56:23.240326 13381 leader_election.cc:302] T 2a7bc6049234402d806ce06fdb8df987 P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 7690ee79577c45e3a2c8f8b552c0bff8; no voters:
I20240503 13:56:23.240393 13184 raft_consensus.cc:3037] T 2a7bc6049234402d806ce06fdb8df987 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:23.240516 14793 raft_consensus.cc:2781] T 2a7bc6049234402d806ce06fdb8df987 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:23.240603 14793 raft_consensus.cc:686] T 2a7bc6049234402d806ce06fdb8df987 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 LEADER]: Becoming Leader. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Running, Role: LEADER
I20240503 13:56:23.240760 14793 consensus_queue.cc:237] T 2a7bc6049234402d806ce06fdb8df987 P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:23.241444 13184 raft_consensus.cc:2454] T 2a7bc6049234402d806ce06fdb8df987 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 1.
I20240503 13:56:23.242650 12912 catalog_manager.cc:5484] T 2a7bc6049234402d806ce06fdb8df987 P 7690ee79577c45e3a2c8f8b552c0bff8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68). New cstate: current_term: 1 leader_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } }
W20240503 13:56:23.366335 14913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:56:23.366571 14913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:56:23.369782 14913 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 13:56:23.369741 14438 consensus_queue.cc:1035] T 97dd13a5495a4ec7818d136e7cc95abd P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:23.374215 14438 consensus_queue.cc:1035] T 97dd13a5495a4ec7818d136e7cc95abd P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_12'
I20240503 13:56:23.463217 13446 raft_consensus.cc:1261] T 7a1c36d4fd0049139c98db7af595bc62 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer 60ba3ffe3b7746469834cd91c84be14e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:23.463315 13314 raft_consensus.cc:1261] T 7a1c36d4fd0049139c98db7af595bc62 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Refusing update from remote peer 60ba3ffe3b7746469834cd91c84be14e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:23.464107 14876 consensus_queue.cc:1035] T 7a1c36d4fd0049139c98db7af595bc62 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:23.465214 13184 raft_consensus.cc:1261] T 2a7bc6049234402d806ce06fdb8df987 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Refusing update from remote peer 7690ee79577c45e3a2c8f8b552c0bff8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:23.465256 14483 consensus_queue.cc:1035] T 7a1c36d4fd0049139c98db7af595bc62 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:23.465757 14911 consensus_queue.cc:1035] T 2a7bc6049234402d806ce06fdb8df987 P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:23.468505 13054 raft_consensus.cc:1261] T 2a7bc6049234402d806ce06fdb8df987 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Refusing update from remote peer 7690ee79577c45e3a2c8f8b552c0bff8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:23.469030 14911 consensus_queue.cc:1035] T 2a7bc6049234402d806ce06fdb8df987 P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:23.476850 14948 mvcc.cc:205] Tried to move back new op lower bound from 7023593813860970496 to 7023593812163186688. Current Snapshot: MvccSnapshot[applied={T|T < 7023593813860970496}]
INSERT report
rows total: 2000
time total: 122.535 ms
time per row: 0.0612676 ms
I20240503 13:56:23.533857 12910 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalance_test_table_13"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:56:23.539291 13295 tablet_service.cc:1467] Processing CreateTablet for tablet 1b714fbd920a447f866b8d4547275675 (DEFAULT_TABLE table=rebalance_test_table_13 [id=4d62f84fd7bc47e8a1f2de384d64b787]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:23.539351 13425 tablet_service.cc:1467] Processing CreateTablet for tablet 1b714fbd920a447f866b8d4547275675 (DEFAULT_TABLE table=rebalance_test_table_13 [id=4d62f84fd7bc47e8a1f2de384d64b787]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:23.539626 13295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b714fbd920a447f866b8d4547275675. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:23.539625 13425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b714fbd920a447f866b8d4547275675. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:23.539706 13426 tablet_service.cc:1467] Processing CreateTablet for tablet b1395d66d8e04ecdafd7a30c50aa1c9a (DEFAULT_TABLE table=rebalance_test_table_13 [id=4d62f84fd7bc47e8a1f2de384d64b787]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:23.539830 13164 tablet_service.cc:1467] Processing CreateTablet for tablet 1b714fbd920a447f866b8d4547275675 (DEFAULT_TABLE table=rebalance_test_table_13 [id=4d62f84fd7bc47e8a1f2de384d64b787]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:23.540001 13426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1395d66d8e04ecdafd7a30c50aa1c9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:23.540055 13164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b714fbd920a447f866b8d4547275675. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:23.542480 14970 tablet_bootstrap.cc:492] T 1b714fbd920a447f866b8d4547275675 P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:23.543359 14970 tablet_bootstrap.cc:654] T 1b714fbd920a447f866b8d4547275675 P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:23.544342 14970 tablet_bootstrap.cc:492] T 1b714fbd920a447f866b8d4547275675 P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:23.544422 14970 ts_tablet_manager.cc:1367] T 1b714fbd920a447f866b8d4547275675 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:23.544889 14970 raft_consensus.cc:348] T 1b714fbd920a447f866b8d4547275675 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:23.545009 14970 raft_consensus.cc:374] T 1b714fbd920a447f866b8d4547275675 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:23.545078 14970 raft_consensus.cc:729] T 1b714fbd920a447f866b8d4547275675 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:23.545229 14970 consensus_queue.cc:260] T 1b714fbd920a447f866b8d4547275675 P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:23.545557 14970 ts_tablet_manager.cc:1398] T 1b714fbd920a447f866b8d4547275675 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:23.548825 13034 tablet_service.cc:1467] Processing CreateTablet for tablet a6a2c05fe3574ab98fb755935b6a92e1 (DEFAULT_TABLE table=rebalance_test_table_13 [id=4d62f84fd7bc47e8a1f2de384d64b787]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:23.549130 13034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6a2c05fe3574ab98fb755935b6a92e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:23.549180 13165 tablet_service.cc:1467] Processing CreateTablet for tablet a6a2c05fe3574ab98fb755935b6a92e1 (DEFAULT_TABLE table=rebalance_test_table_13 [id=4d62f84fd7bc47e8a1f2de384d64b787]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:23.549374 14969 tablet_bootstrap.cc:492] T 1b714fbd920a447f866b8d4547275675 P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:23.549422 13165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6a2c05fe3574ab98fb755935b6a92e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:23.550325 14969 tablet_bootstrap.cc:654] T 1b714fbd920a447f866b8d4547275675 P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:23.551097 14968 tablet_bootstrap.cc:492] T 1b714fbd920a447f866b8d4547275675 P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:23.551633 13295 tablet_service.cc:1467] Processing CreateTablet for tablet b1395d66d8e04ecdafd7a30c50aa1c9a (DEFAULT_TABLE table=rebalance_test_table_13 [id=4d62f84fd7bc47e8a1f2de384d64b787]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:23.551752 14969 tablet_bootstrap.cc:492] T 1b714fbd920a447f866b8d4547275675 P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:23.551829 14969 ts_tablet_manager.cc:1367] T 1b714fbd920a447f866b8d4547275675 P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:56:23.551869 13295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1395d66d8e04ecdafd7a30c50aa1c9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:23.552053 14968 tablet_bootstrap.cc:654] T 1b714fbd920a447f866b8d4547275675 P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:23.552266 14969 raft_consensus.cc:348] T 1b714fbd920a447f866b8d4547275675 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:23.552362 14969 raft_consensus.cc:374] T 1b714fbd920a447f866b8d4547275675 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:23.552387 14969 raft_consensus.cc:729] T 1b714fbd920a447f866b8d4547275675 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:23.552532 14969 consensus_queue.cc:260] T 1b714fbd920a447f866b8d4547275675 P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:23.552727 13296 tablet_service.cc:1467] Processing CreateTablet for tablet a6a2c05fe3574ab98fb755935b6a92e1 (DEFAULT_TABLE table=rebalance_test_table_13 [id=4d62f84fd7bc47e8a1f2de384d64b787]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:23.552788 14969 ts_tablet_manager.cc:1398] T 1b714fbd920a447f866b8d4547275675 P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:23.552963 13296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6a2c05fe3574ab98fb755935b6a92e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:23.553951 14968 tablet_bootstrap.cc:492] T 1b714fbd920a447f866b8d4547275675 P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:23.554031 14968 ts_tablet_manager.cc:1367] T 1b714fbd920a447f866b8d4547275675 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:56:23.554452 14968 raft_consensus.cc:348] T 1b714fbd920a447f866b8d4547275675 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:23.554558 14968 raft_consensus.cc:374] T 1b714fbd920a447f866b8d4547275675 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:23.554587 14968 raft_consensus.cc:729] T 1b714fbd920a447f866b8d4547275675 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:23.554713 14968 consensus_queue.cc:260] T 1b714fbd920a447f866b8d4547275675 P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:23.555051 14968 ts_tablet_manager.cc:1398] T 1b714fbd920a447f866b8d4547275675 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:23.555253 14968 tablet_bootstrap.cc:492] T b1395d66d8e04ecdafd7a30c50aa1c9a P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:23.556787 14968 tablet_bootstrap.cc:654] T b1395d66d8e04ecdafd7a30c50aa1c9a P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:23.557876 14968 tablet_bootstrap.cc:492] T b1395d66d8e04ecdafd7a30c50aa1c9a P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:23.557967 14968 ts_tablet_manager.cc:1367] T b1395d66d8e04ecdafd7a30c50aa1c9a P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:56:23.558387 14968 raft_consensus.cc:348] T b1395d66d8e04ecdafd7a30c50aa1c9a P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:23.558491 14968 raft_consensus.cc:374] T b1395d66d8e04ecdafd7a30c50aa1c9a P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:23.558521 14968 raft_consensus.cc:729] T b1395d66d8e04ecdafd7a30c50aa1c9a P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:23.558624 14968 consensus_queue.cc:260] T b1395d66d8e04ecdafd7a30c50aa1c9a P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:23.558722 13164 tablet_service.cc:1467] Processing CreateTablet for tablet b1395d66d8e04ecdafd7a30c50aa1c9a (DEFAULT_TABLE table=rebalance_test_table_13 [id=4d62f84fd7bc47e8a1f2de384d64b787]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:23.558880 14968 ts_tablet_manager.cc:1398] T b1395d66d8e04ecdafd7a30c50aa1c9a P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:23.558975 13164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1395d66d8e04ecdafd7a30c50aa1c9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:23.560914 14969 tablet_bootstrap.cc:492] T a6a2c05fe3574ab98fb755935b6a92e1 P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:23.561070 14970 tablet_bootstrap.cc:492] T a6a2c05fe3574ab98fb755935b6a92e1 P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:23.561899 14970 tablet_bootstrap.cc:654] T a6a2c05fe3574ab98fb755935b6a92e1 P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:23.564352 14969 tablet_bootstrap.cc:654] T a6a2c05fe3574ab98fb755935b6a92e1 P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:23.564929 14970 tablet_bootstrap.cc:492] T a6a2c05fe3574ab98fb755935b6a92e1 P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:23.565069 14970 ts_tablet_manager.cc:1367] T a6a2c05fe3574ab98fb755935b6a92e1 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:56:23.565505 14970 raft_consensus.cc:348] T a6a2c05fe3574ab98fb755935b6a92e1 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:23.565630 14970 raft_consensus.cc:374] T a6a2c05fe3574ab98fb755935b6a92e1 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:23.565696 14970 raft_consensus.cc:729] T a6a2c05fe3574ab98fb755935b6a92e1 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:23.565798 14969 tablet_bootstrap.cc:492] T a6a2c05fe3574ab98fb755935b6a92e1 P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:23.565845 14970 consensus_queue.cc:260] T a6a2c05fe3574ab98fb755935b6a92e1 P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:23.565943 14969 ts_tablet_manager.cc:1367] T a6a2c05fe3574ab98fb755935b6a92e1 P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240503 13:56:23.566133 14970 ts_tablet_manager.cc:1398] T a6a2c05fe3574ab98fb755935b6a92e1 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:23.566344 14970 tablet_bootstrap.cc:492] T b1395d66d8e04ecdafd7a30c50aa1c9a P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:23.566638 14969 raft_consensus.cc:348] T a6a2c05fe3574ab98fb755935b6a92e1 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:23.566807 14969 raft_consensus.cc:374] T a6a2c05fe3574ab98fb755935b6a92e1 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:23.566898 14969 raft_consensus.cc:729] T a6a2c05fe3574ab98fb755935b6a92e1 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:23.567071 14969 consensus_queue.cc:260] T a6a2c05fe3574ab98fb755935b6a92e1 P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:23.567723 14970 tablet_bootstrap.cc:654] T b1395d66d8e04ecdafd7a30c50aa1c9a P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:23.567957 14969 ts_tablet_manager.cc:1398] T a6a2c05fe3574ab98fb755935b6a92e1 P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:56:23.568166 14969 tablet_bootstrap.cc:492] T b1395d66d8e04ecdafd7a30c50aa1c9a P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:23.568684 14973 tablet_bootstrap.cc:492] T a6a2c05fe3574ab98fb755935b6a92e1 P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:23.569434 14969 tablet_bootstrap.cc:654] T b1395d66d8e04ecdafd7a30c50aa1c9a P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:23.569547 14973 tablet_bootstrap.cc:654] T a6a2c05fe3574ab98fb755935b6a92e1 P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:23.570453 14969 tablet_bootstrap.cc:492] T b1395d66d8e04ecdafd7a30c50aa1c9a P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:23.570577 14969 ts_tablet_manager.cc:1367] T b1395d66d8e04ecdafd7a30c50aa1c9a P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:56:23.570627 14970 tablet_bootstrap.cc:492] T b1395d66d8e04ecdafd7a30c50aa1c9a P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:23.570657 14973 tablet_bootstrap.cc:492] T a6a2c05fe3574ab98fb755935b6a92e1 P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:23.570741 14970 ts_tablet_manager.cc:1367] T b1395d66d8e04ecdafd7a30c50aa1c9a P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:56:23.570802 14973 ts_tablet_manager.cc:1367] T a6a2c05fe3574ab98fb755935b6a92e1 P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:56:23.571034 14969 raft_consensus.cc:348] T b1395d66d8e04ecdafd7a30c50aa1c9a P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:23.571166 14969 raft_consensus.cc:374] T b1395d66d8e04ecdafd7a30c50aa1c9a P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:23.571233 14969 raft_consensus.cc:729] T b1395d66d8e04ecdafd7a30c50aa1c9a P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:23.571300 14970 raft_consensus.cc:348] T b1395d66d8e04ecdafd7a30c50aa1c9a P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:23.571390 14969 consensus_queue.cc:260] T b1395d66d8e04ecdafd7a30c50aa1c9a P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:23.571453 14970 raft_consensus.cc:374] T b1395d66d8e04ecdafd7a30c50aa1c9a P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:23.571487 14970 raft_consensus.cc:729] T b1395d66d8e04ecdafd7a30c50aa1c9a P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:23.571377 14973 raft_consensus.cc:348] T a6a2c05fe3574ab98fb755935b6a92e1 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:23.571530 14973 raft_consensus.cc:374] T a6a2c05fe3574ab98fb755935b6a92e1 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:23.571616 14970 consensus_queue.cc:260] T b1395d66d8e04ecdafd7a30c50aa1c9a P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:23.571653 14973 raft_consensus.cc:729] T a6a2c05fe3574ab98fb755935b6a92e1 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:23.571776 14973 consensus_queue.cc:260] T a6a2c05fe3574ab98fb755935b6a92e1 P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:23.572031 14970 ts_tablet_manager.cc:1398] T b1395d66d8e04ecdafd7a30c50aa1c9a P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:23.572566 14969 ts_tablet_manager.cc:1398] T b1395d66d8e04ecdafd7a30c50aa1c9a P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:23.572743 14973 ts_tablet_manager.cc:1398] T a6a2c05fe3574ab98fb755935b6a92e1 P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:23.577383 14911 raft_consensus.cc:482] T b1395d66d8e04ecdafd7a30c50aa1c9a P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:23.577489 14911 raft_consensus.cc:504] T b1395d66d8e04ecdafd7a30c50aa1c9a P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:23.577843 14911 leader_election.cc:288] T b1395d66d8e04ecdafd7a30c50aa1c9a P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:23.578348 13314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1395d66d8e04ecdafd7a30c50aa1c9a" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703" is_pre_election: true
I20240503 13:56:23.578480 13314 raft_consensus.cc:2454] T b1395d66d8e04ecdafd7a30c50aa1c9a P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 0.
I20240503 13:56:23.578469 13184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1395d66d8e04ecdafd7a30c50aa1c9a" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770" is_pre_election: true
I20240503 13:56:23.578593 13184 raft_consensus.cc:2454] T b1395d66d8e04ecdafd7a30c50aa1c9a P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 0.
I20240503 13:56:23.578759 13383 leader_election.cc:302] T b1395d66d8e04ecdafd7a30c50aa1c9a P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:23.578920 14911 raft_consensus.cc:2781] T b1395d66d8e04ecdafd7a30c50aa1c9a P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:23.578984 14911 raft_consensus.cc:482] T b1395d66d8e04ecdafd7a30c50aa1c9a P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:23.579023 14911 raft_consensus.cc:3037] T b1395d66d8e04ecdafd7a30c50aa1c9a P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:23.579849 14911 raft_consensus.cc:504] T b1395d66d8e04ecdafd7a30c50aa1c9a P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:23.580152 14911 leader_election.cc:288] T b1395d66d8e04ecdafd7a30c50aa1c9a P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 election: Requested vote from peers ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:23.580457 13314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1395d66d8e04ecdafd7a30c50aa1c9a" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703"
I20240503 13:56:23.580475 13184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1395d66d8e04ecdafd7a30c50aa1c9a" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770"
I20240503 13:56:23.580593 13314 raft_consensus.cc:3037] T b1395d66d8e04ecdafd7a30c50aa1c9a P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:23.580605 13184 raft_consensus.cc:3037] T b1395d66d8e04ecdafd7a30c50aa1c9a P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:23.581324 13184 raft_consensus.cc:2454] T b1395d66d8e04ecdafd7a30c50aa1c9a P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 1.
I20240503 13:56:23.581386 13314 raft_consensus.cc:2454] T b1395d66d8e04ecdafd7a30c50aa1c9a P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 1.
I20240503 13:56:23.581591 13383 leader_election.cc:302] T b1395d66d8e04ecdafd7a30c50aa1c9a P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, 8ccf5a4ac450414c9d3579bd5800b770; no voters:
I20240503 13:56:23.581748 14911 raft_consensus.cc:2781] T b1395d66d8e04ecdafd7a30c50aa1c9a P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:23.581836 14911 raft_consensus.cc:686] T b1395d66d8e04ecdafd7a30c50aa1c9a P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 LEADER]: Becoming Leader. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Running, Role: LEADER
I20240503 13:56:23.581965 14911 consensus_queue.cc:237] T b1395d66d8e04ecdafd7a30c50aa1c9a P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:23.583760 12907 catalog_manager.cc:5484] T b1395d66d8e04ecdafd7a30c50aa1c9a P 7690ee79577c45e3a2c8f8b552c0bff8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68). New cstate: current_term: 1 leader_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } }
I20240503 13:56:23.797004 14572 raft_consensus.cc:482] T a6a2c05fe3574ab98fb755935b6a92e1 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:23.797168 14572 raft_consensus.cc:504] T a6a2c05fe3574ab98fb755935b6a92e1 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:23.797621 14572 leader_election.cc:288] T a6a2c05fe3574ab98fb755935b6a92e1 P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695), ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:23.798130 13053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a2c05fe3574ab98fb755935b6a92e1" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e" is_pre_election: true
I20240503 13:56:23.798157 13314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a2c05fe3574ab98fb755935b6a92e1" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703" is_pre_election: true
I20240503 13:56:23.798262 13053 raft_consensus.cc:2454] T a6a2c05fe3574ab98fb755935b6a92e1 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 0.
I20240503 13:56:23.798286 13314 raft_consensus.cc:2454] T a6a2c05fe3574ab98fb755935b6a92e1 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 0.
I20240503 13:56:23.798583 13119 leader_election.cc:302] T a6a2c05fe3574ab98fb755935b6a92e1 P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 8ccf5a4ac450414c9d3579bd5800b770; no voters:
I20240503 13:56:23.798789 14572 raft_consensus.cc:2781] T a6a2c05fe3574ab98fb755935b6a92e1 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:23.798954 14572 raft_consensus.cc:482] T a6a2c05fe3574ab98fb755935b6a92e1 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:23.799048 14572 raft_consensus.cc:3037] T a6a2c05fe3574ab98fb755935b6a92e1 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:23.799849 14572 raft_consensus.cc:504] T a6a2c05fe3574ab98fb755935b6a92e1 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:23.800179 14572 leader_election.cc:288] T a6a2c05fe3574ab98fb755935b6a92e1 P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 election: Requested vote from peers 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695), ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:23.800604 13053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a2c05fe3574ab98fb755935b6a92e1" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e"
I20240503 13:56:23.800604 13314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a2c05fe3574ab98fb755935b6a92e1" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703"
I20240503 13:56:23.800735 13053 raft_consensus.cc:3037] T a6a2c05fe3574ab98fb755935b6a92e1 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:23.800735 13314 raft_consensus.cc:3037] T a6a2c05fe3574ab98fb755935b6a92e1 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:23.801687 13314 raft_consensus.cc:2454] T a6a2c05fe3574ab98fb755935b6a92e1 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 1.
I20240503 13:56:23.801687 13053 raft_consensus.cc:2454] T a6a2c05fe3574ab98fb755935b6a92e1 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 1.
I20240503 13:56:23.801999 13119 leader_election.cc:302] T a6a2c05fe3574ab98fb755935b6a92e1 P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 8ccf5a4ac450414c9d3579bd5800b770; no voters:
I20240503 13:56:23.802173 14572 raft_consensus.cc:2781] T a6a2c05fe3574ab98fb755935b6a92e1 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:23.802248 14572 raft_consensus.cc:686] T a6a2c05fe3574ab98fb755935b6a92e1 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 LEADER]: Becoming Leader. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Running, Role: LEADER
I20240503 13:56:23.802397 14572 consensus_queue.cc:237] T a6a2c05fe3574ab98fb755935b6a92e1 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:23.804339 12907 catalog_manager.cc:5484] T a6a2c05fe3574ab98fb755935b6a92e1 P 8ccf5a4ac450414c9d3579bd5800b770 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66). New cstate: current_term: 1 leader_uuid: "8ccf5a4ac450414c9d3579bd5800b770" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:23.996419 14950 raft_consensus.cc:482] T 1b714fbd920a447f866b8d4547275675 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:23.996599 14950 raft_consensus.cc:504] T 1b714fbd920a447f866b8d4547275675 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:23.996977 14950 leader_election.cc:288] T 1b714fbd920a447f866b8d4547275675 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387), 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:23.997267 13184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b714fbd920a447f866b8d4547275675" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770" is_pre_election: true
I20240503 13:56:23.997291 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b714fbd920a447f866b8d4547275675" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
I20240503 13:56:23.997404 13184 raft_consensus.cc:2454] T 1b714fbd920a447f866b8d4547275675 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 0.
I20240503 13:56:23.997407 13446 raft_consensus.cc:2454] T 1b714fbd920a447f866b8d4547275675 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 0.
I20240503 13:56:23.997689 13249 leader_election.cc:302] T 1b714fbd920a447f866b8d4547275675 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:23.997915 14950 raft_consensus.cc:2781] T 1b714fbd920a447f866b8d4547275675 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:23.997973 14950 raft_consensus.cc:482] T 1b714fbd920a447f866b8d4547275675 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:23.998018 14950 raft_consensus.cc:3037] T 1b714fbd920a447f866b8d4547275675 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:23.998951 14950 raft_consensus.cc:504] T 1b714fbd920a447f866b8d4547275675 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:23.999260 14950 leader_election.cc:288] T 1b714fbd920a447f866b8d4547275675 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: Requested vote from peers 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387), 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:23.999612 13184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b714fbd920a447f866b8d4547275675" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770"
I20240503 13:56:23.999707 13184 raft_consensus.cc:3037] T 1b714fbd920a447f866b8d4547275675 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:24.000398 13184 raft_consensus.cc:2454] T 1b714fbd920a447f866b8d4547275675 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 1.
I20240503 13:56:24.000658 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b714fbd920a447f866b8d4547275675" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
I20240503 13:56:24.000733 13252 leader_election.cc:302] T 1b714fbd920a447f866b8d4547275675 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ccf5a4ac450414c9d3579bd5800b770, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:24.000783 13446 raft_consensus.cc:3037] T 1b714fbd920a447f866b8d4547275675 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:24.000891 14950 raft_consensus.cc:2781] T 1b714fbd920a447f866b8d4547275675 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:24.001039 14950 raft_consensus.cc:686] T 1b714fbd920a447f866b8d4547275675 P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Becoming Leader. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Running, Role: LEADER
I20240503 13:56:24.001178 14950 consensus_queue.cc:237] T 1b714fbd920a447f866b8d4547275675 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:24.001693 13446 raft_consensus.cc:2454] T 1b714fbd920a447f866b8d4547275675 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 1.
I20240503 13:56:24.003015 12907 catalog_manager.cc:5484] T 1b714fbd920a447f866b8d4547275675 P ba0194098aa84449ae17e8a0d4c43703 reported cstate change: term changed from 0 to 1, leader changed from <none> to ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67). New cstate: current_term: 1 leader_uuid: "ba0194098aa84449ae17e8a0d4c43703" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } }
I20240503 13:56:24.135018 14793 consensus_queue.cc:1035] T b1395d66d8e04ecdafd7a30c50aa1c9a P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:24.138697 14911 consensus_queue.cc:1035] T b1395d66d8e04ecdafd7a30c50aa1c9a P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 13:56:24.225601 14985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:56:24.225838 14985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:56:24.229171 14985 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_13'
I20240503 13:56:24.264274 14572 consensus_queue.cc:1035] T a6a2c05fe3574ab98fb755935b6a92e1 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:24.281921 14253 consensus_queue.cc:1035] T a6a2c05fe3574ab98fb755935b6a92e1 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:24.333207 13185 raft_consensus.cc:1261] T 1b714fbd920a447f866b8d4547275675 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:24.333959 13445 raft_consensus.cc:1261] T 1b714fbd920a447f866b8d4547275675 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:24.334117 15028 consensus_queue.cc:1035] T 1b714fbd920a447f866b8d4547275675 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:24.342942 14438 consensus_queue.cc:1035] T 1b714fbd920a447f866b8d4547275675 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 129.674 ms
time per row: 0.0648368 ms
I20240503 13:56:24.399446 12910 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalance_test_table_14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:56:24.405131 13164 tablet_service.cc:1467] Processing CreateTablet for tablet 327a78731e454b0baa59ad7aa54c322b (DEFAULT_TABLE table=rebalance_test_table_14 [id=31d255eaa071464fb137ad403ae1db26]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:24.405215 13426 tablet_service.cc:1467] Processing CreateTablet for tablet 327a78731e454b0baa59ad7aa54c322b (DEFAULT_TABLE table=rebalance_test_table_14 [id=31d255eaa071464fb137ad403ae1db26]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:24.405488 13164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 327a78731e454b0baa59ad7aa54c322b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:24.405568 13426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 327a78731e454b0baa59ad7aa54c322b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:24.405509 13034 tablet_service.cc:1467] Processing CreateTablet for tablet 327a78731e454b0baa59ad7aa54c322b (DEFAULT_TABLE table=rebalance_test_table_14 [id=31d255eaa071464fb137ad403ae1db26]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:24.405769 13034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 327a78731e454b0baa59ad7aa54c322b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:24.405735 13033 tablet_service.cc:1467] Processing CreateTablet for tablet d4e32f627d314080b985a65eddc851dd (DEFAULT_TABLE table=rebalance_test_table_14 [id=31d255eaa071464fb137ad403ae1db26]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:24.405972 13033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4e32f627d314080b985a65eddc851dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:24.408042 13032 tablet_service.cc:1467] Processing CreateTablet for tablet 866a258ecb2e4685908d5382cb594959 (DEFAULT_TABLE table=rebalance_test_table_14 [id=31d255eaa071464fb137ad403ae1db26]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:24.408278 13032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 866a258ecb2e4685908d5382cb594959. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:24.408517 15046 tablet_bootstrap.cc:492] T 327a78731e454b0baa59ad7aa54c322b P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:24.409119 13295 tablet_service.cc:1467] Processing CreateTablet for tablet d4e32f627d314080b985a65eddc851dd (DEFAULT_TABLE table=rebalance_test_table_14 [id=31d255eaa071464fb137ad403ae1db26]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:24.409384 13295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4e32f627d314080b985a65eddc851dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:24.409466 15046 tablet_bootstrap.cc:654] T 327a78731e454b0baa59ad7aa54c322b P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:24.410671 13426 tablet_service.cc:1467] Processing CreateTablet for tablet d4e32f627d314080b985a65eddc851dd (DEFAULT_TABLE table=rebalance_test_table_14 [id=31d255eaa071464fb137ad403ae1db26]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:24.410741 13425 tablet_service.cc:1467] Processing CreateTablet for tablet 866a258ecb2e4685908d5382cb594959 (DEFAULT_TABLE table=rebalance_test_table_14 [id=31d255eaa071464fb137ad403ae1db26]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:24.410832 13164 tablet_service.cc:1467] Processing CreateTablet for tablet 866a258ecb2e4685908d5382cb594959 (DEFAULT_TABLE table=rebalance_test_table_14 [id=31d255eaa071464fb137ad403ae1db26]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:24.410905 13426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4e32f627d314080b985a65eddc851dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:24.411044 13164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 866a258ecb2e4685908d5382cb594959. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:24.411067 13425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 866a258ecb2e4685908d5382cb594959. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:24.413105 15046 tablet_bootstrap.cc:492] T 327a78731e454b0baa59ad7aa54c322b P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:24.413110 15050 tablet_bootstrap.cc:492] T d4e32f627d314080b985a65eddc851dd P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:24.413265 15046 ts_tablet_manager.cc:1367] T 327a78731e454b0baa59ad7aa54c322b P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:56:24.413480 15048 tablet_bootstrap.cc:492] T d4e32f627d314080b985a65eddc851dd P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:24.413846 15046 raft_consensus.cc:348] T 327a78731e454b0baa59ad7aa54c322b P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:24.413129 15047 tablet_bootstrap.cc:492] T 327a78731e454b0baa59ad7aa54c322b P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:24.413962 15050 tablet_bootstrap.cc:654] T d4e32f627d314080b985a65eddc851dd P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:24.413991 15046 raft_consensus.cc:374] T 327a78731e454b0baa59ad7aa54c322b P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:24.414045 15046 raft_consensus.cc:729] T 327a78731e454b0baa59ad7aa54c322b P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:24.414193 15046 consensus_queue.cc:260] T 327a78731e454b0baa59ad7aa54c322b P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:24.414543 15046 ts_tablet_manager.cc:1398] T 327a78731e454b0baa59ad7aa54c322b P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:24.414757 15046 tablet_bootstrap.cc:492] T 866a258ecb2e4685908d5382cb594959 P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:24.414848 15047 tablet_bootstrap.cc:654] T 327a78731e454b0baa59ad7aa54c322b P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:24.415268 15048 tablet_bootstrap.cc:654] T d4e32f627d314080b985a65eddc851dd P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:24.415633 15046 tablet_bootstrap.cc:654] T 866a258ecb2e4685908d5382cb594959 P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:24.416649 15047 tablet_bootstrap.cc:492] T 327a78731e454b0baa59ad7aa54c322b P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:24.416649 15050 tablet_bootstrap.cc:492] T d4e32f627d314080b985a65eddc851dd P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:24.416734 15050 ts_tablet_manager.cc:1367] T d4e32f627d314080b985a65eddc851dd P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:56:24.416734 15047 ts_tablet_manager.cc:1367] T 327a78731e454b0baa59ad7aa54c322b P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:56:24.417174 15047 raft_consensus.cc:348] T 327a78731e454b0baa59ad7aa54c322b P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:24.417220 15050 raft_consensus.cc:348] T d4e32f627d314080b985a65eddc851dd P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:24.417273 15047 raft_consensus.cc:374] T 327a78731e454b0baa59ad7aa54c322b P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:24.417302 15047 raft_consensus.cc:729] T 327a78731e454b0baa59ad7aa54c322b P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:24.417305 15050 raft_consensus.cc:374] T d4e32f627d314080b985a65eddc851dd P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:24.417335 15050 raft_consensus.cc:729] T d4e32f627d314080b985a65eddc851dd P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:24.417438 15047 consensus_queue.cc:260] T 327a78731e454b0baa59ad7aa54c322b P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:24.417469 15050 consensus_queue.cc:260] T d4e32f627d314080b985a65eddc851dd P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:24.417716 15050 ts_tablet_manager.cc:1398] T d4e32f627d314080b985a65eddc851dd P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:24.417716 15047 ts_tablet_manager.cc:1398] T 327a78731e454b0baa59ad7aa54c322b P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:24.418025 15047 tablet_bootstrap.cc:492] T d4e32f627d314080b985a65eddc851dd P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:24.418323 15048 tablet_bootstrap.cc:492] T d4e32f627d314080b985a65eddc851dd P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:24.418406 15048 ts_tablet_manager.cc:1367] T d4e32f627d314080b985a65eddc851dd P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 13:56:24.418845 15047 tablet_bootstrap.cc:654] T d4e32f627d314080b985a65eddc851dd P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:24.418778 15048 raft_consensus.cc:348] T d4e32f627d314080b985a65eddc851dd P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:24.418890 15048 raft_consensus.cc:374] T d4e32f627d314080b985a65eddc851dd P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:24.418920 15048 raft_consensus.cc:729] T d4e32f627d314080b985a65eddc851dd P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:24.419049 15048 consensus_queue.cc:260] T d4e32f627d314080b985a65eddc851dd P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:24.419308 15048 ts_tablet_manager.cc:1398] T d4e32f627d314080b985a65eddc851dd P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:24.419487 15048 tablet_bootstrap.cc:492] T 327a78731e454b0baa59ad7aa54c322b P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:24.419812 15047 tablet_bootstrap.cc:492] T d4e32f627d314080b985a65eddc851dd P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:24.419867 15047 ts_tablet_manager.cc:1367] T d4e32f627d314080b985a65eddc851dd P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:24.420255 15048 tablet_bootstrap.cc:654] T 327a78731e454b0baa59ad7aa54c322b P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:24.421248 15048 tablet_bootstrap.cc:492] T 327a78731e454b0baa59ad7aa54c322b P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:24.421247 15047 raft_consensus.cc:348] T d4e32f627d314080b985a65eddc851dd P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:24.421329 15048 ts_tablet_manager.cc:1367] T 327a78731e454b0baa59ad7aa54c322b P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:56:24.421341 15047 raft_consensus.cc:374] T d4e32f627d314080b985a65eddc851dd P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:24.421365 15047 raft_consensus.cc:729] T d4e32f627d314080b985a65eddc851dd P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:24.421484 15047 consensus_queue.cc:260] T d4e32f627d314080b985a65eddc851dd P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:24.421708 15048 raft_consensus.cc:348] T 327a78731e454b0baa59ad7aa54c322b P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:24.421773 15047 ts_tablet_manager.cc:1398] T d4e32f627d314080b985a65eddc851dd P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:24.421802 15048 raft_consensus.cc:374] T 327a78731e454b0baa59ad7aa54c322b P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:24.421833 15048 raft_consensus.cc:729] T 327a78731e454b0baa59ad7aa54c322b P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:24.421952 15048 consensus_queue.cc:260] T 327a78731e454b0baa59ad7aa54c322b P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:24.422202 15048 ts_tablet_manager.cc:1398] T 327a78731e454b0baa59ad7aa54c322b P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:24.422291 15047 tablet_bootstrap.cc:492] T 866a258ecb2e4685908d5382cb594959 P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:24.422549 15048 tablet_bootstrap.cc:492] T 866a258ecb2e4685908d5382cb594959 P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:24.423256 15047 tablet_bootstrap.cc:654] T 866a258ecb2e4685908d5382cb594959 P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:24.423360 15048 tablet_bootstrap.cc:654] T 866a258ecb2e4685908d5382cb594959 P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:24.425176 15048 tablet_bootstrap.cc:492] T 866a258ecb2e4685908d5382cb594959 P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:24.425253 15048 ts_tablet_manager.cc:1367] T 866a258ecb2e4685908d5382cb594959 P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:56:24.425495 15047 tablet_bootstrap.cc:492] T 866a258ecb2e4685908d5382cb594959 P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:24.425561 15047 ts_tablet_manager.cc:1367] T 866a258ecb2e4685908d5382cb594959 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:56:24.425637 15048 raft_consensus.cc:348] T 866a258ecb2e4685908d5382cb594959 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:24.425722 15048 raft_consensus.cc:374] T 866a258ecb2e4685908d5382cb594959 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:24.425748 15048 raft_consensus.cc:729] T 866a258ecb2e4685908d5382cb594959 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:24.425886 15046 tablet_bootstrap.cc:492] T 866a258ecb2e4685908d5382cb594959 P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:24.425993 15046 ts_tablet_manager.cc:1367] T 866a258ecb2e4685908d5382cb594959 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 13:56:24.425977 15047 raft_consensus.cc:348] T 866a258ecb2e4685908d5382cb594959 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:24.426075 15047 raft_consensus.cc:374] T 866a258ecb2e4685908d5382cb594959 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:24.426103 15047 raft_consensus.cc:729] T 866a258ecb2e4685908d5382cb594959 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:24.426213 15047 consensus_queue.cc:260] T 866a258ecb2e4685908d5382cb594959 P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:24.426386 15046 raft_consensus.cc:348] T 866a258ecb2e4685908d5382cb594959 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:24.426481 15046 raft_consensus.cc:374] T 866a258ecb2e4685908d5382cb594959 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:24.426508 15046 raft_consensus.cc:729] T 866a258ecb2e4685908d5382cb594959 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:24.426499 15047 ts_tablet_manager.cc:1398] T 866a258ecb2e4685908d5382cb594959 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:24.426508 15048 consensus_queue.cc:260] T 866a258ecb2e4685908d5382cb594959 P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:24.426621 15046 consensus_queue.cc:260] T 866a258ecb2e4685908d5382cb594959 P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:24.426903 15046 ts_tablet_manager.cc:1398] T 866a258ecb2e4685908d5382cb594959 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:24.426935 15048 ts_tablet_manager.cc:1398] T 866a258ecb2e4685908d5382cb594959 P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:24.483000 14950 raft_consensus.cc:482] T d4e32f627d314080b985a65eddc851dd P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:24.483114 14950 raft_consensus.cc:504] T d4e32f627d314080b985a65eddc851dd P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:24.483462 14950 leader_election.cc:288] T d4e32f627d314080b985a65eddc851dd P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695), 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:24.483855 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4e32f627d314080b985a65eddc851dd" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
I20240503 13:56:24.483860 13054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4e32f627d314080b985a65eddc851dd" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e" is_pre_election: true
I20240503 13:56:24.483978 13054 raft_consensus.cc:2454] T d4e32f627d314080b985a65eddc851dd P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 0.
I20240503 13:56:24.483983 13445 raft_consensus.cc:2454] T d4e32f627d314080b985a65eddc851dd P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 0.
I20240503 13:56:24.484294 13250 leader_election.cc:302] T d4e32f627d314080b985a65eddc851dd P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:24.484467 14950 raft_consensus.cc:2781] T d4e32f627d314080b985a65eddc851dd P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:24.484550 14950 raft_consensus.cc:482] T d4e32f627d314080b985a65eddc851dd P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:24.484606 14950 raft_consensus.cc:3037] T d4e32f627d314080b985a65eddc851dd P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:24.485422 14950 raft_consensus.cc:504] T d4e32f627d314080b985a65eddc851dd P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:24.485743 14950 leader_election.cc:288] T d4e32f627d314080b985a65eddc851dd P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: Requested vote from peers 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695), 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:24.486063 13054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4e32f627d314080b985a65eddc851dd" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e"
I20240503 13:56:24.486094 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4e32f627d314080b985a65eddc851dd" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
I20240503 13:56:24.486186 13054 raft_consensus.cc:3037] T d4e32f627d314080b985a65eddc851dd P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:24.486193 13445 raft_consensus.cc:3037] T d4e32f627d314080b985a65eddc851dd P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:24.486932 13445 raft_consensus.cc:2454] T d4e32f627d314080b985a65eddc851dd P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 1.
I20240503 13:56:24.486974 13054 raft_consensus.cc:2454] T d4e32f627d314080b985a65eddc851dd P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 1.
I20240503 13:56:24.487242 13250 leader_election.cc:302] T d4e32f627d314080b985a65eddc851dd P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:24.487414 14950 raft_consensus.cc:2781] T d4e32f627d314080b985a65eddc851dd P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:24.487497 14950 raft_consensus.cc:686] T d4e32f627d314080b985a65eddc851dd P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Becoming Leader. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Running, Role: LEADER
I20240503 13:56:24.487632 14950 consensus_queue.cc:237] T d4e32f627d314080b985a65eddc851dd P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:24.489449 12906 catalog_manager.cc:5484] T d4e32f627d314080b985a65eddc851dd P ba0194098aa84449ae17e8a0d4c43703 reported cstate change: term changed from 0 to 1, leader changed from <none> to ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67). New cstate: current_term: 1 leader_uuid: "ba0194098aa84449ae17e8a0d4c43703" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:24.491310 15044 raft_consensus.cc:482] T 327a78731e454b0baa59ad7aa54c322b P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:24.491416 15044 raft_consensus.cc:504] T 327a78731e454b0baa59ad7aa54c322b P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:24.491866 15044 leader_election.cc:288] T 327a78731e454b0baa59ad7aa54c322b P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:24.492192 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "327a78731e454b0baa59ad7aa54c322b" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
I20240503 13:56:24.492213 13054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "327a78731e454b0baa59ad7aa54c322b" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e" is_pre_election: true
I20240503 13:56:24.492331 13054 raft_consensus.cc:2454] T 327a78731e454b0baa59ad7aa54c322b P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 0.
I20240503 13:56:24.492326 13445 raft_consensus.cc:2454] T 327a78731e454b0baa59ad7aa54c322b P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 0.
I20240503 13:56:24.492601 13118 leader_election.cc:302] T 327a78731e454b0baa59ad7aa54c322b P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, 8ccf5a4ac450414c9d3579bd5800b770; no voters:
I20240503 13:56:24.492763 15044 raft_consensus.cc:2781] T 327a78731e454b0baa59ad7aa54c322b P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:24.492827 15044 raft_consensus.cc:482] T 327a78731e454b0baa59ad7aa54c322b P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:24.492873 15044 raft_consensus.cc:3037] T 327a78731e454b0baa59ad7aa54c322b P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:24.493628 15044 raft_consensus.cc:504] T 327a78731e454b0baa59ad7aa54c322b P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:24.493927 15044 leader_election.cc:288] T 327a78731e454b0baa59ad7aa54c322b P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 election: Requested vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:24.494230 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "327a78731e454b0baa59ad7aa54c322b" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
I20240503 13:56:24.494262 13054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "327a78731e454b0baa59ad7aa54c322b" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e"
I20240503 13:56:24.494346 13445 raft_consensus.cc:3037] T 327a78731e454b0baa59ad7aa54c322b P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:24.494346 13054 raft_consensus.cc:3037] T 327a78731e454b0baa59ad7aa54c322b P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:24.495239 13445 raft_consensus.cc:2454] T 327a78731e454b0baa59ad7aa54c322b P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 1.
I20240503 13:56:24.495239 13054 raft_consensus.cc:2454] T 327a78731e454b0baa59ad7aa54c322b P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 1.
I20240503 13:56:24.495523 13119 leader_election.cc:302] T 327a78731e454b0baa59ad7aa54c322b P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 8ccf5a4ac450414c9d3579bd5800b770; no voters:
I20240503 13:56:24.495673 15044 raft_consensus.cc:2781] T 327a78731e454b0baa59ad7aa54c322b P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:24.495764 15044 raft_consensus.cc:686] T 327a78731e454b0baa59ad7aa54c322b P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 LEADER]: Becoming Leader. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Running, Role: LEADER
I20240503 13:56:24.495895 15044 consensus_queue.cc:237] T 327a78731e454b0baa59ad7aa54c322b P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:24.497794 12910 catalog_manager.cc:5484] T 327a78731e454b0baa59ad7aa54c322b P 8ccf5a4ac450414c9d3579bd5800b770 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66). New cstate: current_term: 1 leader_uuid: "8ccf5a4ac450414c9d3579bd5800b770" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } }
I20240503 13:56:24.540083 14793 raft_consensus.cc:482] T 866a258ecb2e4685908d5382cb594959 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:24.540200 14793 raft_consensus.cc:504] T 866a258ecb2e4685908d5382cb594959 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:24.540577 14793 leader_election.cc:288] T 866a258ecb2e4685908d5382cb594959 P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:24.540939 13184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "866a258ecb2e4685908d5382cb594959" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770" is_pre_election: true
I20240503 13:56:24.540939 13054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "866a258ecb2e4685908d5382cb594959" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e" is_pre_election: true
I20240503 13:56:24.541102 13054 raft_consensus.cc:2454] T 866a258ecb2e4685908d5382cb594959 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 0.
I20240503 13:56:24.541112 13184 raft_consensus.cc:2454] T 866a258ecb2e4685908d5382cb594959 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 0.
I20240503 13:56:24.541378 13383 leader_election.cc:302] T 866a258ecb2e4685908d5382cb594959 P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, 8ccf5a4ac450414c9d3579bd5800b770; no voters:
I20240503 13:56:24.541600 14793 raft_consensus.cc:2781] T 866a258ecb2e4685908d5382cb594959 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:24.541651 14793 raft_consensus.cc:482] T 866a258ecb2e4685908d5382cb594959 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:24.541676 14793 raft_consensus.cc:3037] T 866a258ecb2e4685908d5382cb594959 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:24.542603 14793 raft_consensus.cc:504] T 866a258ecb2e4685908d5382cb594959 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:24.542932 14793 leader_election.cc:288] T 866a258ecb2e4685908d5382cb594959 P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 election: Requested vote from peers 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:24.543195 13054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "866a258ecb2e4685908d5382cb594959" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e"
I20240503 13:56:24.543315 13054 raft_consensus.cc:3037] T 866a258ecb2e4685908d5382cb594959 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:24.543262 13184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "866a258ecb2e4685908d5382cb594959" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770"
I20240503 13:56:24.543390 13184 raft_consensus.cc:3037] T 866a258ecb2e4685908d5382cb594959 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:24.544278 13184 raft_consensus.cc:2454] T 866a258ecb2e4685908d5382cb594959 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 1.
I20240503 13:56:24.544278 13054 raft_consensus.cc:2454] T 866a258ecb2e4685908d5382cb594959 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 1.
I20240503 13:56:24.544610 13383 leader_election.cc:302] T 866a258ecb2e4685908d5382cb594959 P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, 8ccf5a4ac450414c9d3579bd5800b770; no voters:
I20240503 13:56:24.545964 14911 raft_consensus.cc:2781] T 866a258ecb2e4685908d5382cb594959 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:24.546120 14911 raft_consensus.cc:686] T 866a258ecb2e4685908d5382cb594959 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 LEADER]: Becoming Leader. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Running, Role: LEADER
I20240503 13:56:24.546255 14911 consensus_queue.cc:237] T 866a258ecb2e4685908d5382cb594959 P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:24.548203 12910 catalog_manager.cc:5484] T 866a258ecb2e4685908d5382cb594959 P 7690ee79577c45e3a2c8f8b552c0bff8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68). New cstate: current_term: 1 leader_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } }
W20240503 13:56:24.677816 15063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:56:24.678086 15063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:56:24.681138 15063 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_14'
I20240503 13:56:24.768771 13445 raft_consensus.cc:1261] T d4e32f627d314080b985a65eddc851dd P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:24.769181 14950 consensus_queue.cc:1035] T d4e32f627d314080b985a65eddc851dd P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:24.772202 13054 raft_consensus.cc:1261] T 866a258ecb2e4685908d5382cb594959 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Refusing update from remote peer 7690ee79577c45e3a2c8f8b552c0bff8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:24.772239 13184 raft_consensus.cc:1261] T 866a258ecb2e4685908d5382cb594959 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Refusing update from remote peer 7690ee79577c45e3a2c8f8b552c0bff8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:24.772640 14911 consensus_queue.cc:1035] T 866a258ecb2e4685908d5382cb594959 P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:24.775242 13054 raft_consensus.cc:1261] T d4e32f627d314080b985a65eddc851dd P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:24.775673 15061 consensus_queue.cc:1035] T 866a258ecb2e4685908d5382cb594959 P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:24.777382 13053 raft_consensus.cc:1261] T 327a78731e454b0baa59ad7aa54c322b P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Refusing update from remote peer 8ccf5a4ac450414c9d3579bd5800b770: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:24.778285 14253 consensus_queue.cc:1035] T 327a78731e454b0baa59ad7aa54c322b P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:24.778713 15028 consensus_queue.cc:1035] T d4e32f627d314080b985a65eddc851dd P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:24.788956 13445 raft_consensus.cc:1261] T 327a78731e454b0baa59ad7aa54c322b P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer 8ccf5a4ac450414c9d3579bd5800b770: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:24.789462 14253 consensus_queue.cc:1035] T 327a78731e454b0baa59ad7aa54c322b P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:24.792454 15099 mvcc.cc:205] Tried to move back new op lower bound from 7023593819221700608 to 7023593818302791680. Current Snapshot: MvccSnapshot[applied={T|T < 7023593819221700608 or (T in {7023593819221700608})}]
INSERT report
rows total: 2000
time total: 115.692 ms
time per row: 0.0578459 ms
I20240503 13:56:24.840119 12911 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalance_test_table_15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:56:24.846462 13296 tablet_service.cc:1467] Processing CreateTablet for tablet b34b432bb82e465f87a6a3b05d7adfd2 (DEFAULT_TABLE table=rebalance_test_table_15 [id=2bf18d88eb674480901ad018abd82ad1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:24.846632 13295 tablet_service.cc:1467] Processing CreateTablet for tablet 4939335096b94444a61503841e2e81eb (DEFAULT_TABLE table=rebalance_test_table_15 [id=2bf18d88eb674480901ad018abd82ad1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:24.846800 13296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b34b432bb82e465f87a6a3b05d7adfd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:24.846823 13032 tablet_service.cc:1467] Processing CreateTablet for tablet 4939335096b94444a61503841e2e81eb (DEFAULT_TABLE table=rebalance_test_table_15 [id=2bf18d88eb674480901ad018abd82ad1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:24.847019 13295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4939335096b94444a61503841e2e81eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:24.847131 13032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4939335096b94444a61503841e2e81eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:24.849043 15048 tablet_bootstrap.cc:492] T 4939335096b94444a61503841e2e81eb P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:24.849224 15050 tablet_bootstrap.cc:492] T 4939335096b94444a61503841e2e81eb P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:24.846326 13164 tablet_service.cc:1467] Processing CreateTablet for tablet 4939335096b94444a61503841e2e81eb (DEFAULT_TABLE table=rebalance_test_table_15 [id=2bf18d88eb674480901ad018abd82ad1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:24.849910 13164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4939335096b94444a61503841e2e81eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:24.850108 15050 tablet_bootstrap.cc:654] T 4939335096b94444a61503841e2e81eb P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:24.850306 15048 tablet_bootstrap.cc:654] T 4939335096b94444a61503841e2e81eb P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:24.851305 13425 tablet_service.cc:1467] Processing CreateTablet for tablet b34b432bb82e465f87a6a3b05d7adfd2 (DEFAULT_TABLE table=rebalance_test_table_15 [id=2bf18d88eb674480901ad018abd82ad1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:24.851469 15050 tablet_bootstrap.cc:492] T 4939335096b94444a61503841e2e81eb P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:24.851557 15050 ts_tablet_manager.cc:1367] T 4939335096b94444a61503841e2e81eb P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:24.851568 13425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b34b432bb82e465f87a6a3b05d7adfd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:24.851994 15050 raft_consensus.cc:348] T 4939335096b94444a61503841e2e81eb P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:24.852088 15046 tablet_bootstrap.cc:492] T 4939335096b94444a61503841e2e81eb P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:24.852092 15050 raft_consensus.cc:374] T 4939335096b94444a61503841e2e81eb P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:24.852121 15050 raft_consensus.cc:729] T 4939335096b94444a61503841e2e81eb P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:24.849607 13165 tablet_service.cc:1467] Processing CreateTablet for tablet 397f0e3e877d4acd9157bdf99bee6255 (DEFAULT_TABLE table=rebalance_test_table_15 [id=2bf18d88eb674480901ad018abd82ad1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:24.852232 15050 consensus_queue.cc:260] T 4939335096b94444a61503841e2e81eb P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:24.852277 13165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 397f0e3e877d4acd9157bdf99bee6255. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:24.852519 15050 ts_tablet_manager.cc:1398] T 4939335096b94444a61503841e2e81eb P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:24.853410 15046 tablet_bootstrap.cc:654] T 4939335096b94444a61503841e2e81eb P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:24.853787 13295 tablet_service.cc:1467] Processing CreateTablet for tablet 397f0e3e877d4acd9157bdf99bee6255 (DEFAULT_TABLE table=rebalance_test_table_15 [id=2bf18d88eb674480901ad018abd82ad1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:24.853870 15047 tablet_bootstrap.cc:492] T b34b432bb82e465f87a6a3b05d7adfd2 P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:24.854008 13295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 397f0e3e877d4acd9157bdf99bee6255. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:24.854678 15047 tablet_bootstrap.cc:654] T b34b432bb82e465f87a6a3b05d7adfd2 P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:24.855683 15047 tablet_bootstrap.cc:492] T b34b432bb82e465f87a6a3b05d7adfd2 P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:24.855773 15047 ts_tablet_manager.cc:1367] T b34b432bb82e465f87a6a3b05d7adfd2 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:56:24.855819 13032 tablet_service.cc:1467] Processing CreateTablet for tablet b34b432bb82e465f87a6a3b05d7adfd2 (DEFAULT_TABLE table=rebalance_test_table_15 [id=2bf18d88eb674480901ad018abd82ad1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:24.856108 13032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b34b432bb82e465f87a6a3b05d7adfd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:24.856220 15047 raft_consensus.cc:348] T b34b432bb82e465f87a6a3b05d7adfd2 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:24.856335 15047 raft_consensus.cc:374] T b34b432bb82e465f87a6a3b05d7adfd2 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:24.856459 15047 raft_consensus.cc:729] T b34b432bb82e465f87a6a3b05d7adfd2 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:24.856559 13425 tablet_service.cc:1467] Processing CreateTablet for tablet 397f0e3e877d4acd9157bdf99bee6255 (DEFAULT_TABLE table=rebalance_test_table_15 [id=2bf18d88eb674480901ad018abd82ad1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:24.856647 15047 consensus_queue.cc:260] T b34b432bb82e465f87a6a3b05d7adfd2 P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:24.856777 13425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 397f0e3e877d4acd9157bdf99bee6255. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:24.856799 15050 tablet_bootstrap.cc:492] T b34b432bb82e465f87a6a3b05d7adfd2 P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:24.857028 15047 ts_tablet_manager.cc:1398] T b34b432bb82e465f87a6a3b05d7adfd2 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:24.857707 15050 tablet_bootstrap.cc:654] T b34b432bb82e465f87a6a3b05d7adfd2 P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:24.858781 15050 tablet_bootstrap.cc:492] T b34b432bb82e465f87a6a3b05d7adfd2 P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:24.858863 15050 ts_tablet_manager.cc:1367] T b34b432bb82e465f87a6a3b05d7adfd2 P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:56:24.858932 15046 tablet_bootstrap.cc:492] T 4939335096b94444a61503841e2e81eb P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:24.859033 15046 ts_tablet_manager.cc:1367] T 4939335096b94444a61503841e2e81eb P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240503 13:56:24.859045 15047 tablet_bootstrap.cc:492] T 397f0e3e877d4acd9157bdf99bee6255 P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:24.859308 15050 raft_consensus.cc:348] T b34b432bb82e465f87a6a3b05d7adfd2 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:24.859431 15050 raft_consensus.cc:374] T b34b432bb82e465f87a6a3b05d7adfd2 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:24.859462 15050 raft_consensus.cc:729] T b34b432bb82e465f87a6a3b05d7adfd2 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:24.859465 15046 raft_consensus.cc:348] T 4939335096b94444a61503841e2e81eb P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:24.859570 15046 raft_consensus.cc:374] T 4939335096b94444a61503841e2e81eb P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:24.859565 15050 consensus_queue.cc:260] T b34b432bb82e465f87a6a3b05d7adfd2 P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:24.859602 15046 raft_consensus.cc:729] T 4939335096b94444a61503841e2e81eb P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:24.859719 15046 consensus_queue.cc:260] T 4939335096b94444a61503841e2e81eb P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:24.859833 15050 ts_tablet_manager.cc:1398] T b34b432bb82e465f87a6a3b05d7adfd2 P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:24.859961 15047 tablet_bootstrap.cc:654] T 397f0e3e877d4acd9157bdf99bee6255 P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:24.860002 15050 tablet_bootstrap.cc:492] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:24.860590 15046 ts_tablet_manager.cc:1398] T 4939335096b94444a61503841e2e81eb P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:56:24.860790 15046 tablet_bootstrap.cc:492] T 397f0e3e877d4acd9157bdf99bee6255 P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:24.860878 15050 tablet_bootstrap.cc:654] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:24.861675 15046 tablet_bootstrap.cc:654] T 397f0e3e877d4acd9157bdf99bee6255 P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:24.861888 15050 tablet_bootstrap.cc:492] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:24.861969 15050 ts_tablet_manager.cc:1367] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:56:24.863351 15050 raft_consensus.cc:348] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:24.863466 15050 raft_consensus.cc:374] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:24.863492 15050 raft_consensus.cc:729] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:24.863605 15050 consensus_queue.cc:260] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:24.863881 15050 ts_tablet_manager.cc:1398] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:56:24.864795 15048 tablet_bootstrap.cc:492] T 4939335096b94444a61503841e2e81eb P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:24.864907 15048 ts_tablet_manager.cc:1367] T 4939335096b94444a61503841e2e81eb P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240503 13:56:24.864971 15047 tablet_bootstrap.cc:492] T 397f0e3e877d4acd9157bdf99bee6255 P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:24.865061 15047 ts_tablet_manager.cc:1367] T 397f0e3e877d4acd9157bdf99bee6255 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240503 13:56:24.865409 15048 raft_consensus.cc:348] T 4939335096b94444a61503841e2e81eb P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:24.865459 15047 raft_consensus.cc:348] T 397f0e3e877d4acd9157bdf99bee6255 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:24.865540 15048 raft_consensus.cc:374] T 4939335096b94444a61503841e2e81eb P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:24.865545 15047 raft_consensus.cc:374] T 397f0e3e877d4acd9157bdf99bee6255 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:24.865573 15047 raft_consensus.cc:729] T 397f0e3e877d4acd9157bdf99bee6255 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:24.865573 15048 raft_consensus.cc:729] T 4939335096b94444a61503841e2e81eb P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:24.865725 15047 consensus_queue.cc:260] T 397f0e3e877d4acd9157bdf99bee6255 P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:24.865854 15048 consensus_queue.cc:260] T 4939335096b94444a61503841e2e81eb P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:24.866011 15047 ts_tablet_manager.cc:1398] T 397f0e3e877d4acd9157bdf99bee6255 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:24.866156 15048 ts_tablet_manager.cc:1398] T 4939335096b94444a61503841e2e81eb P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:24.866876 15048 tablet_bootstrap.cc:492] T b34b432bb82e465f87a6a3b05d7adfd2 P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:24.867342 15046 tablet_bootstrap.cc:492] T 397f0e3e877d4acd9157bdf99bee6255 P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:24.867430 15046 ts_tablet_manager.cc:1367] T 397f0e3e877d4acd9157bdf99bee6255 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240503 13:56:24.867816 15046 raft_consensus.cc:348] T 397f0e3e877d4acd9157bdf99bee6255 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:24.867911 15046 raft_consensus.cc:374] T 397f0e3e877d4acd9157bdf99bee6255 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:24.867940 15046 raft_consensus.cc:729] T 397f0e3e877d4acd9157bdf99bee6255 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:24.867956 15048 tablet_bootstrap.cc:654] T b34b432bb82e465f87a6a3b05d7adfd2 P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:24.868135 15046 consensus_queue.cc:260] T 397f0e3e877d4acd9157bdf99bee6255 P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:24.868520 15046 ts_tablet_manager.cc:1398] T 397f0e3e877d4acd9157bdf99bee6255 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:24.869204 15048 tablet_bootstrap.cc:492] T b34b432bb82e465f87a6a3b05d7adfd2 P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:24.869946 15048 ts_tablet_manager.cc:1367] T b34b432bb82e465f87a6a3b05d7adfd2 P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 13:56:24.870431 15048 raft_consensus.cc:348] T b34b432bb82e465f87a6a3b05d7adfd2 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:24.870596 15048 raft_consensus.cc:374] T b34b432bb82e465f87a6a3b05d7adfd2 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:24.870674 15048 raft_consensus.cc:729] T b34b432bb82e465f87a6a3b05d7adfd2 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:24.870940 15048 consensus_queue.cc:260] T b34b432bb82e465f87a6a3b05d7adfd2 P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:24.871336 15048 ts_tablet_manager.cc:1398] T b34b432bb82e465f87a6a3b05d7adfd2 P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:24.900563 15061 raft_consensus.cc:482] T b34b432bb82e465f87a6a3b05d7adfd2 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:24.900671 15061 raft_consensus.cc:504] T b34b432bb82e465f87a6a3b05d7adfd2 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:24.901074 15061 leader_election.cc:288] T b34b432bb82e465f87a6a3b05d7adfd2 P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279), 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:24.901407 13314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b34b432bb82e465f87a6a3b05d7adfd2" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703" is_pre_election: true
I20240503 13:56:24.901535 13314 raft_consensus.cc:2454] T b34b432bb82e465f87a6a3b05d7adfd2 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 0.
I20240503 13:56:24.901664 13053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b34b432bb82e465f87a6a3b05d7adfd2" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e" is_pre_election: true
I20240503 13:56:24.901799 13053 raft_consensus.cc:2454] T b34b432bb82e465f87a6a3b05d7adfd2 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 0.
I20240503 13:56:24.901851 13383 leader_election.cc:302] T b34b432bb82e465f87a6a3b05d7adfd2 P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:24.902052 15061 raft_consensus.cc:2781] T b34b432bb82e465f87a6a3b05d7adfd2 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:24.902110 15061 raft_consensus.cc:482] T b34b432bb82e465f87a6a3b05d7adfd2 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:24.902145 15061 raft_consensus.cc:3037] T b34b432bb82e465f87a6a3b05d7adfd2 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:24.903065 15061 raft_consensus.cc:504] T b34b432bb82e465f87a6a3b05d7adfd2 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:24.903362 15061 leader_election.cc:288] T b34b432bb82e465f87a6a3b05d7adfd2 P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 election: Requested vote from peers ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279), 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:24.903643 13314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b34b432bb82e465f87a6a3b05d7adfd2" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703"
I20240503 13:56:24.903676 13053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b34b432bb82e465f87a6a3b05d7adfd2" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e"
I20240503 13:56:24.903744 13314 raft_consensus.cc:3037] T b34b432bb82e465f87a6a3b05d7adfd2 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:24.903766 13053 raft_consensus.cc:3037] T b34b432bb82e465f87a6a3b05d7adfd2 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:24.904448 13314 raft_consensus.cc:2454] T b34b432bb82e465f87a6a3b05d7adfd2 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 1.
I20240503 13:56:24.904639 13053 raft_consensus.cc:2454] T b34b432bb82e465f87a6a3b05d7adfd2 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 1.
I20240503 13:56:24.904770 13383 leader_election.cc:302] T b34b432bb82e465f87a6a3b05d7adfd2 P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:24.904918 15061 raft_consensus.cc:2781] T b34b432bb82e465f87a6a3b05d7adfd2 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:24.905090 15061 raft_consensus.cc:686] T b34b432bb82e465f87a6a3b05d7adfd2 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 LEADER]: Becoming Leader. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Running, Role: LEADER
I20240503 13:56:24.905232 15061 consensus_queue.cc:237] T b34b432bb82e465f87a6a3b05d7adfd2 P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:24.907153 12910 catalog_manager.cc:5484] T b34b432bb82e465f87a6a3b05d7adfd2 P 7690ee79577c45e3a2c8f8b552c0bff8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68). New cstate: current_term: 1 leader_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } }
I20240503 13:56:24.963762 14950 raft_consensus.cc:482] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:24.963904 14950 raft_consensus.cc:504] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:24.964351 14950 leader_election.cc:288] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387), 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:24.964732 13184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "397f0e3e877d4acd9157bdf99bee6255" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770" is_pre_election: true
I20240503 13:56:24.964789 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "397f0e3e877d4acd9157bdf99bee6255" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
I20240503 13:56:24.964919 13445 raft_consensus.cc:2454] T 397f0e3e877d4acd9157bdf99bee6255 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 0.
I20240503 13:56:24.964919 13184 raft_consensus.cc:2454] T 397f0e3e877d4acd9157bdf99bee6255 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 0.
I20240503 13:56:24.965209 13252 leader_election.cc:302] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ccf5a4ac450414c9d3579bd5800b770, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:24.965373 14950 raft_consensus.cc:2781] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:24.965432 14950 raft_consensus.cc:482] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:24.965495 14950 raft_consensus.cc:3037] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:24.966269 14950 raft_consensus.cc:504] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:24.966641 14950 leader_election.cc:288] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: Requested vote from peers 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387), 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:24.966975 13184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "397f0e3e877d4acd9157bdf99bee6255" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770"
I20240503 13:56:24.967119 13184 raft_consensus.cc:3037] T 397f0e3e877d4acd9157bdf99bee6255 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:24.967638 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "397f0e3e877d4acd9157bdf99bee6255" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
I20240503 13:56:24.967759 13445 raft_consensus.cc:3037] T 397f0e3e877d4acd9157bdf99bee6255 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:24.968144 13184 raft_consensus.cc:2454] T 397f0e3e877d4acd9157bdf99bee6255 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 1.
I20240503 13:56:24.968454 13252 leader_election.cc:302] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ccf5a4ac450414c9d3579bd5800b770, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:24.968647 14950 raft_consensus.cc:2781] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:24.968735 14950 raft_consensus.cc:686] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Becoming Leader. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Running, Role: LEADER
I20240503 13:56:24.968832 13445 raft_consensus.cc:2454] T 397f0e3e877d4acd9157bdf99bee6255 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 1.
I20240503 13:56:24.968873 14950 consensus_queue.cc:237] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:24.971252 12910 catalog_manager.cc:5484] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 reported cstate change: term changed from 0 to 1, leader changed from <none> to ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67). New cstate: current_term: 1 leader_uuid: "ba0194098aa84449ae17e8a0d4c43703" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } }
I20240503 13:56:24.971486 14950 raft_consensus.cc:482] T 4939335096b94444a61503841e2e81eb P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:24.971568 14950 raft_consensus.cc:504] T 4939335096b94444a61503841e2e81eb P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:24.972002 14950 leader_election.cc:288] T 4939335096b94444a61503841e2e81eb P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387), 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:24.972893 13053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4939335096b94444a61503841e2e81eb" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e" is_pre_election: true
I20240503 13:56:24.973026 13053 raft_consensus.cc:2454] T 4939335096b94444a61503841e2e81eb P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 0.
I20240503 13:56:24.973827 13250 leader_election.cc:302] T 4939335096b94444a61503841e2e81eb P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:24.973968 14950 raft_consensus.cc:2781] T 4939335096b94444a61503841e2e81eb P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:24.974013 14950 raft_consensus.cc:482] T 4939335096b94444a61503841e2e81eb P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:24.974048 14950 raft_consensus.cc:3037] T 4939335096b94444a61503841e2e81eb P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:24.974332 13184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4939335096b94444a61503841e2e81eb" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770" is_pre_election: true
I20240503 13:56:24.974486 13184 raft_consensus.cc:2454] T 4939335096b94444a61503841e2e81eb P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 0.
I20240503 13:56:24.975034 14950 raft_consensus.cc:504] T 4939335096b94444a61503841e2e81eb P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:24.975354 14950 leader_election.cc:288] T 4939335096b94444a61503841e2e81eb P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: Requested vote from peers 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387), 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:24.975654 13184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4939335096b94444a61503841e2e81eb" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770"
I20240503 13:56:24.975688 13053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4939335096b94444a61503841e2e81eb" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e"
I20240503 13:56:24.975771 13184 raft_consensus.cc:3037] T 4939335096b94444a61503841e2e81eb P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:24.975790 13053 raft_consensus.cc:3037] T 4939335096b94444a61503841e2e81eb P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:24.976716 13053 raft_consensus.cc:2454] T 4939335096b94444a61503841e2e81eb P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 1.
I20240503 13:56:24.976716 13184 raft_consensus.cc:2454] T 4939335096b94444a61503841e2e81eb P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 1.
I20240503 13:56:24.977011 13250 leader_election.cc:302] T 4939335096b94444a61503841e2e81eb P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:24.977160 14950 raft_consensus.cc:2781] T 4939335096b94444a61503841e2e81eb P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:24.977258 14950 raft_consensus.cc:686] T 4939335096b94444a61503841e2e81eb P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Becoming Leader. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Running, Role: LEADER
I20240503 13:56:24.977398 14950 consensus_queue.cc:237] T 4939335096b94444a61503841e2e81eb P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:24.979389 12910 catalog_manager.cc:5484] T 4939335096b94444a61503841e2e81eb P ba0194098aa84449ae17e8a0d4c43703 reported cstate change: term changed from 0 to 1, leader changed from <none> to ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67). New cstate: current_term: 1 leader_uuid: "ba0194098aa84449ae17e8a0d4c43703" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } }
W20240503 13:56:25.119432 15117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:56:25.119671 15117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:56:25.122749 15117 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_15'
I20240503 13:56:25.208747 13053 raft_consensus.cc:1261] T 4939335096b94444a61503841e2e81eb P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:25.211983 13184 raft_consensus.cc:1261] T 4939335096b94444a61503841e2e81eb P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:25.212543 13445 raft_consensus.cc:1261] T 397f0e3e877d4acd9157bdf99bee6255 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:25.212782 15028 consensus_queue.cc:1035] T 4939335096b94444a61503841e2e81eb P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:25.213011 14950 consensus_queue.cc:1035] T 4939335096b94444a61503841e2e81eb P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:25.213065 15028 consensus_queue.cc:1035] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:25.213805 13314 raft_consensus.cc:1261] T b34b432bb82e465f87a6a3b05d7adfd2 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Refusing update from remote peer 7690ee79577c45e3a2c8f8b552c0bff8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:25.213928 13053 raft_consensus.cc:1261] T b34b432bb82e465f87a6a3b05d7adfd2 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Refusing update from remote peer 7690ee79577c45e3a2c8f8b552c0bff8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:25.214179 15061 consensus_queue.cc:1035] T b34b432bb82e465f87a6a3b05d7adfd2 P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:25.214303 14911 consensus_queue.cc:1035] T b34b432bb82e465f87a6a3b05d7adfd2 P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:25.223855 13185 raft_consensus.cc:1261] T 397f0e3e877d4acd9157bdf99bee6255 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:25.226049 15143 mvcc.cc:205] Tried to move back new op lower bound from 7023593821006995456 to 7023593820068540416. Current Snapshot: MvccSnapshot[applied={T|T < 7023593821006995456}]
I20240503 13:56:25.229434 15028 consensus_queue.cc:1035] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 115.677 ms
time per row: 0.0578383 ms
I20240503 13:56:25.279002 12909 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalance_test_table_16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:56:25.284538 13165 tablet_service.cc:1467] Processing CreateTablet for tablet 2cb96587807b4437ba9b1b1ca5c95a6a (DEFAULT_TABLE table=rebalance_test_table_16 [id=8817f3bcbcde485f96984e3c87bf3241]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:25.284672 13032 tablet_service.cc:1467] Processing CreateTablet for tablet 2cb96587807b4437ba9b1b1ca5c95a6a (DEFAULT_TABLE table=rebalance_test_table_16 [id=8817f3bcbcde485f96984e3c87bf3241]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:25.284881 13165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cb96587807b4437ba9b1b1ca5c95a6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:25.284897 13032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cb96587807b4437ba9b1b1ca5c95a6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:25.285193 13425 tablet_service.cc:1467] Processing CreateTablet for tablet 9f4d9dca210846edaf7f9bc818c5db29 (DEFAULT_TABLE table=rebalance_test_table_16 [id=8817f3bcbcde485f96984e3c87bf3241]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:25.285293 13426 tablet_service.cc:1467] Processing CreateTablet for tablet e9b5dc0762494e6ebdebb7b2833a272e (DEFAULT_TABLE table=rebalance_test_table_16 [id=8817f3bcbcde485f96984e3c87bf3241]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:25.285472 13425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f4d9dca210846edaf7f9bc818c5db29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:25.285622 13426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9b5dc0762494e6ebdebb7b2833a272e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:25.287045 15048 tablet_bootstrap.cc:492] T 2cb96587807b4437ba9b1b1ca5c95a6a P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:25.287473 15046 tablet_bootstrap.cc:492] T 2cb96587807b4437ba9b1b1ca5c95a6a P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:25.287909 15048 tablet_bootstrap.cc:654] T 2cb96587807b4437ba9b1b1ca5c95a6a P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:25.288185 15046 tablet_bootstrap.cc:654] T 2cb96587807b4437ba9b1b1ca5c95a6a P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:25.288753 15047 tablet_bootstrap.cc:492] T 9f4d9dca210846edaf7f9bc818c5db29 P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:25.288976 15048 tablet_bootstrap.cc:492] T 2cb96587807b4437ba9b1b1ca5c95a6a P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:25.289073 15048 ts_tablet_manager.cc:1367] T 2cb96587807b4437ba9b1b1ca5c95a6a P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 13:56:25.289382 13295 tablet_service.cc:1467] Processing CreateTablet for tablet 2cb96587807b4437ba9b1b1ca5c95a6a (DEFAULT_TABLE table=rebalance_test_table_16 [id=8817f3bcbcde485f96984e3c87bf3241]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:25.289443 13296 tablet_service.cc:1467] Processing CreateTablet for tablet e9b5dc0762494e6ebdebb7b2833a272e (DEFAULT_TABLE table=rebalance_test_table_16 [id=8817f3bcbcde485f96984e3c87bf3241]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:25.289489 15048 raft_consensus.cc:348] T 2cb96587807b4437ba9b1b1ca5c95a6a P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:25.289556 15047 tablet_bootstrap.cc:654] T 9f4d9dca210846edaf7f9bc818c5db29 P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:25.289582 15048 raft_consensus.cc:374] T 2cb96587807b4437ba9b1b1ca5c95a6a P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:25.289613 15048 raft_consensus.cc:729] T 2cb96587807b4437ba9b1b1ca5c95a6a P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:25.289641 13295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cb96587807b4437ba9b1b1ca5c95a6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:25.289726 15048 consensus_queue.cc:260] T 2cb96587807b4437ba9b1b1ca5c95a6a P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:25.289912 13165 tablet_service.cc:1467] Processing CreateTablet for tablet 9f4d9dca210846edaf7f9bc818c5db29 (DEFAULT_TABLE table=rebalance_test_table_16 [id=8817f3bcbcde485f96984e3c87bf3241]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:25.290014 15048 ts_tablet_manager.cc:1398] T 2cb96587807b4437ba9b1b1ca5c95a6a P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:25.290138 13165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f4d9dca210846edaf7f9bc818c5db29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:25.290971 13033 tablet_service.cc:1467] Processing CreateTablet for tablet e9b5dc0762494e6ebdebb7b2833a272e (DEFAULT_TABLE table=rebalance_test_table_16 [id=8817f3bcbcde485f96984e3c87bf3241]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:25.291111 13296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9b5dc0762494e6ebdebb7b2833a272e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:25.291196 13033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9b5dc0762494e6ebdebb7b2833a272e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:25.293061 15050 tablet_bootstrap.cc:492] T 2cb96587807b4437ba9b1b1ca5c95a6a P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:25.293712 13032 tablet_service.cc:1467] Processing CreateTablet for tablet 9f4d9dca210846edaf7f9bc818c5db29 (DEFAULT_TABLE table=rebalance_test_table_16 [id=8817f3bcbcde485f96984e3c87bf3241]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:25.293820 15050 tablet_bootstrap.cc:654] T 2cb96587807b4437ba9b1b1ca5c95a6a P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:25.293912 13032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f4d9dca210846edaf7f9bc818c5db29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:25.295624 15050 tablet_bootstrap.cc:492] T 2cb96587807b4437ba9b1b1ca5c95a6a P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:25.295712 15050 ts_tablet_manager.cc:1367] T 2cb96587807b4437ba9b1b1ca5c95a6a P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 13:56:25.296139 15050 raft_consensus.cc:348] T 2cb96587807b4437ba9b1b1ca5c95a6a P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:25.296272 15050 raft_consensus.cc:374] T 2cb96587807b4437ba9b1b1ca5c95a6a P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:25.296315 15050 raft_consensus.cc:729] T 2cb96587807b4437ba9b1b1ca5c95a6a P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:25.296459 15050 consensus_queue.cc:260] T 2cb96587807b4437ba9b1b1ca5c95a6a P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:25.296752 15050 ts_tablet_manager.cc:1398] T 2cb96587807b4437ba9b1b1ca5c95a6a P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:25.296926 15050 tablet_bootstrap.cc:492] T e9b5dc0762494e6ebdebb7b2833a272e P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:25.297122 15047 tablet_bootstrap.cc:492] T 9f4d9dca210846edaf7f9bc818c5db29 P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:25.297204 15047 ts_tablet_manager.cc:1367] T 9f4d9dca210846edaf7f9bc818c5db29 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20240503 13:56:25.297662 15050 tablet_bootstrap.cc:654] T e9b5dc0762494e6ebdebb7b2833a272e P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:25.298254 15046 tablet_bootstrap.cc:492] T 2cb96587807b4437ba9b1b1ca5c95a6a P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:25.298321 15048 tablet_bootstrap.cc:492] T e9b5dc0762494e6ebdebb7b2833a272e P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:25.298339 15046 ts_tablet_manager.cc:1367] T 2cb96587807b4437ba9b1b1ca5c95a6a P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20240503 13:56:25.298767 15046 raft_consensus.cc:348] T 2cb96587807b4437ba9b1b1ca5c95a6a P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:25.298864 15046 raft_consensus.cc:374] T 2cb96587807b4437ba9b1b1ca5c95a6a P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:25.298893 15046 raft_consensus.cc:729] T 2cb96587807b4437ba9b1b1ca5c95a6a P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:25.299028 15046 consensus_queue.cc:260] T 2cb96587807b4437ba9b1b1ca5c95a6a P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:25.299113 15048 tablet_bootstrap.cc:654] T e9b5dc0762494e6ebdebb7b2833a272e P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:25.299280 15050 tablet_bootstrap.cc:492] T e9b5dc0762494e6ebdebb7b2833a272e P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:25.299289 15046 ts_tablet_manager.cc:1398] T 2cb96587807b4437ba9b1b1ca5c95a6a P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:25.299348 15050 ts_tablet_manager.cc:1367] T e9b5dc0762494e6ebdebb7b2833a272e P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:25.299729 15050 raft_consensus.cc:348] T e9b5dc0762494e6ebdebb7b2833a272e P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:25.299798 15050 raft_consensus.cc:374] T e9b5dc0762494e6ebdebb7b2833a272e P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:25.299815 15050 raft_consensus.cc:729] T e9b5dc0762494e6ebdebb7b2833a272e P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:25.299945 15050 consensus_queue.cc:260] T e9b5dc0762494e6ebdebb7b2833a272e P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:25.300345 15050 ts_tablet_manager.cc:1398] T e9b5dc0762494e6ebdebb7b2833a272e P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:25.297636 15047 raft_consensus.cc:348] T 9f4d9dca210846edaf7f9bc818c5db29 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:25.300411 15047 raft_consensus.cc:374] T 9f4d9dca210846edaf7f9bc818c5db29 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:25.300457 15047 raft_consensus.cc:729] T 9f4d9dca210846edaf7f9bc818c5db29 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:25.300613 15047 consensus_queue.cc:260] T 9f4d9dca210846edaf7f9bc818c5db29 P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:25.300880 15047 ts_tablet_manager.cc:1398] T 9f4d9dca210846edaf7f9bc818c5db29 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 13:56:25.301007 15048 tablet_bootstrap.cc:492] T e9b5dc0762494e6ebdebb7b2833a272e P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:25.301035 15047 tablet_bootstrap.cc:492] T e9b5dc0762494e6ebdebb7b2833a272e P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:25.301081 15048 ts_tablet_manager.cc:1367] T e9b5dc0762494e6ebdebb7b2833a272e P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 13:56:25.301474 15048 raft_consensus.cc:348] T e9b5dc0762494e6ebdebb7b2833a272e P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:25.301570 15048 raft_consensus.cc:374] T e9b5dc0762494e6ebdebb7b2833a272e P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:25.301601 15048 raft_consensus.cc:729] T e9b5dc0762494e6ebdebb7b2833a272e P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:25.301728 15048 consensus_queue.cc:260] T e9b5dc0762494e6ebdebb7b2833a272e P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:25.301798 15047 tablet_bootstrap.cc:654] T e9b5dc0762494e6ebdebb7b2833a272e P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:25.303474 15046 tablet_bootstrap.cc:492] T 9f4d9dca210846edaf7f9bc818c5db29 P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:25.304294 15046 tablet_bootstrap.cc:654] T 9f4d9dca210846edaf7f9bc818c5db29 P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:25.305473 15047 tablet_bootstrap.cc:492] T e9b5dc0762494e6ebdebb7b2833a272e P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:25.305665 15047 ts_tablet_manager.cc:1367] T e9b5dc0762494e6ebdebb7b2833a272e P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 13:56:25.306069 15047 raft_consensus.cc:348] T e9b5dc0762494e6ebdebb7b2833a272e P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:25.306146 15047 raft_consensus.cc:374] T e9b5dc0762494e6ebdebb7b2833a272e P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:25.306165 15047 raft_consensus.cc:729] T e9b5dc0762494e6ebdebb7b2833a272e P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:25.306262 15047 consensus_queue.cc:260] T e9b5dc0762494e6ebdebb7b2833a272e P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:25.306521 15047 ts_tablet_manager.cc:1398] T e9b5dc0762494e6ebdebb7b2833a272e P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:25.307106 15046 tablet_bootstrap.cc:492] T 9f4d9dca210846edaf7f9bc818c5db29 P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:25.307189 15046 ts_tablet_manager.cc:1367] T 9f4d9dca210846edaf7f9bc818c5db29 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:56:25.307816 15046 raft_consensus.cc:348] T 9f4d9dca210846edaf7f9bc818c5db29 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:25.308096 15046 raft_consensus.cc:374] T 9f4d9dca210846edaf7f9bc818c5db29 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:25.308148 15046 raft_consensus.cc:729] T 9f4d9dca210846edaf7f9bc818c5db29 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:25.308259 15046 consensus_queue.cc:260] T 9f4d9dca210846edaf7f9bc818c5db29 P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:25.309015 15046 ts_tablet_manager.cc:1398] T 9f4d9dca210846edaf7f9bc818c5db29 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:25.312517 15048 ts_tablet_manager.cc:1398] T e9b5dc0762494e6ebdebb7b2833a272e P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
I20240503 13:56:25.313294 15048 tablet_bootstrap.cc:492] T 9f4d9dca210846edaf7f9bc818c5db29 P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:25.314415 15048 tablet_bootstrap.cc:654] T 9f4d9dca210846edaf7f9bc818c5db29 P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:25.315508 15048 tablet_bootstrap.cc:492] T 9f4d9dca210846edaf7f9bc818c5db29 P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:25.315600 15048 ts_tablet_manager.cc:1367] T 9f4d9dca210846edaf7f9bc818c5db29 P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:56:25.316076 15048 raft_consensus.cc:348] T 9f4d9dca210846edaf7f9bc818c5db29 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:25.316190 15048 raft_consensus.cc:374] T 9f4d9dca210846edaf7f9bc818c5db29 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:25.316223 15048 raft_consensus.cc:729] T 9f4d9dca210846edaf7f9bc818c5db29 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:25.316350 15048 consensus_queue.cc:260] T 9f4d9dca210846edaf7f9bc818c5db29 P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:25.316752 15048 ts_tablet_manager.cc:1398] T 9f4d9dca210846edaf7f9bc818c5db29 P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:25.455336 15096 raft_consensus.cc:482] T e9b5dc0762494e6ebdebb7b2833a272e P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:25.455462 15096 raft_consensus.cc:504] T e9b5dc0762494e6ebdebb7b2833a272e P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:25.455850 15096 leader_election.cc:288] T e9b5dc0762494e6ebdebb7b2833a272e P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279), 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:25.456210 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9b5dc0762494e6ebdebb7b2833a272e" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
I20240503 13:56:25.456211 13314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9b5dc0762494e6ebdebb7b2833a272e" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703" is_pre_election: true
I20240503 13:56:25.456358 13314 raft_consensus.cc:2454] T e9b5dc0762494e6ebdebb7b2833a272e P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 0.
I20240503 13:56:25.456362 13447 raft_consensus.cc:2454] T e9b5dc0762494e6ebdebb7b2833a272e P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 0.
I20240503 13:56:25.456643 12990 leader_election.cc:302] T e9b5dc0762494e6ebdebb7b2833a272e P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:25.456817 15096 raft_consensus.cc:2781] T e9b5dc0762494e6ebdebb7b2833a272e P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:25.456871 15096 raft_consensus.cc:482] T e9b5dc0762494e6ebdebb7b2833a272e P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:25.456894 15096 raft_consensus.cc:3037] T e9b5dc0762494e6ebdebb7b2833a272e P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:25.457815 15096 raft_consensus.cc:504] T e9b5dc0762494e6ebdebb7b2833a272e P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:25.458173 15096 leader_election.cc:288] T e9b5dc0762494e6ebdebb7b2833a272e P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 election: Requested vote from peers ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279), 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:25.458451 13314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9b5dc0762494e6ebdebb7b2833a272e" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703"
I20240503 13:56:25.458463 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9b5dc0762494e6ebdebb7b2833a272e" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
I20240503 13:56:25.458551 13314 raft_consensus.cc:3037] T e9b5dc0762494e6ebdebb7b2833a272e P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:25.458560 13447 raft_consensus.cc:3037] T e9b5dc0762494e6ebdebb7b2833a272e P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:25.459267 13314 raft_consensus.cc:2454] T e9b5dc0762494e6ebdebb7b2833a272e P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 1.
I20240503 13:56:25.459424 13447 raft_consensus.cc:2454] T e9b5dc0762494e6ebdebb7b2833a272e P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 1.
I20240503 13:56:25.459546 12990 leader_election.cc:302] T e9b5dc0762494e6ebdebb7b2833a272e P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:25.459757 15096 raft_consensus.cc:2781] T e9b5dc0762494e6ebdebb7b2833a272e P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:25.459830 15096 raft_consensus.cc:686] T e9b5dc0762494e6ebdebb7b2833a272e P 60ba3ffe3b7746469834cd91c84be14e [term 1 LEADER]: Becoming Leader. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Running, Role: LEADER
I20240503 13:56:25.459951 15096 consensus_queue.cc:237] T e9b5dc0762494e6ebdebb7b2833a272e P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:25.461763 12909 catalog_manager.cc:5484] T e9b5dc0762494e6ebdebb7b2833a272e P 60ba3ffe3b7746469834cd91c84be14e reported cstate change: term changed from 0 to 1, leader changed from <none> to 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65). New cstate: current_term: 1 leader_uuid: "60ba3ffe3b7746469834cd91c84be14e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } }
I20240503 13:56:25.560617 15061 raft_consensus.cc:482] T 9f4d9dca210846edaf7f9bc818c5db29 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:25.560736 15061 raft_consensus.cc:504] T 9f4d9dca210846edaf7f9bc818c5db29 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:25.561116 15061 leader_election.cc:288] T 9f4d9dca210846edaf7f9bc818c5db29 P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387), 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:25.561434 13183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f4d9dca210846edaf7f9bc818c5db29" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770" is_pre_election: true
I20240503 13:56:25.561441 13053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f4d9dca210846edaf7f9bc818c5db29" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e" is_pre_election: true
I20240503 13:56:25.561582 13053 raft_consensus.cc:2454] T 9f4d9dca210846edaf7f9bc818c5db29 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 0.
I20240503 13:56:25.561582 13183 raft_consensus.cc:2454] T 9f4d9dca210846edaf7f9bc818c5db29 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 0.
I20240503 13:56:25.561858 13381 leader_election.cc:302] T 9f4d9dca210846edaf7f9bc818c5db29 P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 7690ee79577c45e3a2c8f8b552c0bff8; no voters:
I20240503 13:56:25.562069 15061 raft_consensus.cc:2781] T 9f4d9dca210846edaf7f9bc818c5db29 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:25.562124 15061 raft_consensus.cc:482] T 9f4d9dca210846edaf7f9bc818c5db29 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:25.562168 15061 raft_consensus.cc:3037] T 9f4d9dca210846edaf7f9bc818c5db29 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:25.562968 15061 raft_consensus.cc:504] T 9f4d9dca210846edaf7f9bc818c5db29 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:25.563308 15061 leader_election.cc:288] T 9f4d9dca210846edaf7f9bc818c5db29 P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 election: Requested vote from peers 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387), 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:25.563439 15152 raft_consensus.cc:482] T 2cb96587807b4437ba9b1b1ca5c95a6a P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:25.563550 15152 raft_consensus.cc:504] T 2cb96587807b4437ba9b1b1ca5c95a6a P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:25.563928 15152 leader_election.cc:288] T 2cb96587807b4437ba9b1b1ca5c95a6a P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387), 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:25.564093 13053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f4d9dca210846edaf7f9bc818c5db29" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e"
I20240503 13:56:25.564205 13053 raft_consensus.cc:3037] T 9f4d9dca210846edaf7f9bc818c5db29 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:25.565176 13053 raft_consensus.cc:2454] T 9f4d9dca210846edaf7f9bc818c5db29 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 1.
I20240503 13:56:25.565464 13381 leader_election.cc:302] T 9f4d9dca210846edaf7f9bc818c5db29 P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 7690ee79577c45e3a2c8f8b552c0bff8; no voters:
I20240503 13:56:25.565601 15061 raft_consensus.cc:2781] T 9f4d9dca210846edaf7f9bc818c5db29 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:25.565558 13053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cb96587807b4437ba9b1b1ca5c95a6a" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e" is_pre_election: true
I20240503 13:56:25.565673 15061 raft_consensus.cc:686] T 9f4d9dca210846edaf7f9bc818c5db29 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 LEADER]: Becoming Leader. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Running, Role: LEADER
I20240503 13:56:25.565685 13053 raft_consensus.cc:2454] T 2cb96587807b4437ba9b1b1ca5c95a6a P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 0.
I20240503 13:56:25.565806 15061 consensus_queue.cc:237] T 9f4d9dca210846edaf7f9bc818c5db29 P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:25.566565 13250 leader_election.cc:302] T 2cb96587807b4437ba9b1b1ca5c95a6a P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:25.566732 15152 raft_consensus.cc:2781] T 2cb96587807b4437ba9b1b1ca5c95a6a P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:25.566777 15152 raft_consensus.cc:482] T 2cb96587807b4437ba9b1b1ca5c95a6a P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:25.566810 15152 raft_consensus.cc:3037] T 2cb96587807b4437ba9b1b1ca5c95a6a P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:25.567700 15152 raft_consensus.cc:504] T 2cb96587807b4437ba9b1b1ca5c95a6a P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:25.568024 15152 leader_election.cc:288] T 2cb96587807b4437ba9b1b1ca5c95a6a P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: Requested vote from peers 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387), 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:25.567961 12909 catalog_manager.cc:5484] T 9f4d9dca210846edaf7f9bc818c5db29 P 7690ee79577c45e3a2c8f8b552c0bff8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68). New cstate: current_term: 1 leader_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } }
I20240503 13:56:25.568688 13052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cb96587807b4437ba9b1b1ca5c95a6a" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e"
I20240503 13:56:25.568722 13183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cb96587807b4437ba9b1b1ca5c95a6a" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770" is_pre_election: true
I20240503 13:56:25.568804 13052 raft_consensus.cc:3037] T 2cb96587807b4437ba9b1b1ca5c95a6a P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:25.568835 13183 raft_consensus.cc:2454] T 2cb96587807b4437ba9b1b1ca5c95a6a P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 0.
I20240503 13:56:25.569150 13183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f4d9dca210846edaf7f9bc818c5db29" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770"
I20240503 13:56:25.569244 13183 raft_consensus.cc:3037] T 9f4d9dca210846edaf7f9bc818c5db29 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:25.569706 13052 raft_consensus.cc:2454] T 2cb96587807b4437ba9b1b1ca5c95a6a P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 1.
I20240503 13:56:25.570080 13250 leader_election.cc:302] T 2cb96587807b4437ba9b1b1ca5c95a6a P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:25.570171 13183 raft_consensus.cc:2454] T 9f4d9dca210846edaf7f9bc818c5db29 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 1.
I20240503 13:56:25.570580 15152 raft_consensus.cc:2781] T 2cb96587807b4437ba9b1b1ca5c95a6a P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:25.570834 15152 raft_consensus.cc:686] T 2cb96587807b4437ba9b1b1ca5c95a6a P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Becoming Leader. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Running, Role: LEADER
I20240503 13:56:25.570863 13183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cb96587807b4437ba9b1b1ca5c95a6a" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770"
I20240503 13:56:25.570945 13183 raft_consensus.cc:3037] T 2cb96587807b4437ba9b1b1ca5c95a6a P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:25.570986 15152 consensus_queue.cc:237] T 2cb96587807b4437ba9b1b1ca5c95a6a P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:25.571679 13183 raft_consensus.cc:2454] T 2cb96587807b4437ba9b1b1ca5c95a6a P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 1.
I20240503 13:56:25.572789 12909 catalog_manager.cc:5484] T 2cb96587807b4437ba9b1b1ca5c95a6a P ba0194098aa84449ae17e8a0d4c43703 reported cstate change: term changed from 0 to 1, leader changed from <none> to ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67). New cstate: current_term: 1 leader_uuid: "ba0194098aa84449ae17e8a0d4c43703" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
W20240503 13:56:25.708297 15169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:56:25.708634 15169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:56:25.711836 15169 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_16'
I20240503 13:56:25.808073 13447 raft_consensus.cc:1261] T e9b5dc0762494e6ebdebb7b2833a272e P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer 60ba3ffe3b7746469834cd91c84be14e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:25.808715 13314 raft_consensus.cc:1261] T e9b5dc0762494e6ebdebb7b2833a272e P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Refusing update from remote peer 60ba3ffe3b7746469834cd91c84be14e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:25.809208 15096 consensus_queue.cc:1035] T e9b5dc0762494e6ebdebb7b2833a272e P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:25.810317 15096 consensus_queue.cc:1035] T e9b5dc0762494e6ebdebb7b2833a272e P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:25.816112 13183 raft_consensus.cc:1261] T 9f4d9dca210846edaf7f9bc818c5db29 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Refusing update from remote peer 7690ee79577c45e3a2c8f8b552c0bff8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:25.816509 13184 raft_consensus.cc:1261] T 2cb96587807b4437ba9b1b1ca5c95a6a P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:25.816949 13053 raft_consensus.cc:1261] T 2cb96587807b4437ba9b1b1ca5c95a6a P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:25.817641 15154 consensus_queue.cc:1035] T 2cb96587807b4437ba9b1b1ca5c95a6a P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:25.817939 15154 consensus_queue.cc:1035] T 2cb96587807b4437ba9b1b1ca5c95a6a P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:25.818363 14911 consensus_queue.cc:1035] T 9f4d9dca210846edaf7f9bc818c5db29 P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:25.819340 13053 raft_consensus.cc:1261] T 9f4d9dca210846edaf7f9bc818c5db29 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Refusing update from remote peer 7690ee79577c45e3a2c8f8b552c0bff8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:25.821779 14911 consensus_queue.cc:1035] T 9f4d9dca210846edaf7f9bc818c5db29 P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 132.267 ms
time per row: 0.0661333 ms
I20240503 13:56:25.888726 12909 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalance_test_table_17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:56:25.894901 13426 tablet_service.cc:1467] Processing CreateTablet for tablet 3bb148fcf60f410089420a9499f08086 (DEFAULT_TABLE table=rebalance_test_table_17 [id=52f6b247496e4c3eb63eef4f16160012]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:25.895229 13426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bb148fcf60f410089420a9499f08086. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:25.895390 13165 tablet_service.cc:1467] Processing CreateTablet for tablet 3bb148fcf60f410089420a9499f08086 (DEFAULT_TABLE table=rebalance_test_table_17 [id=52f6b247496e4c3eb63eef4f16160012]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:25.895632 13165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bb148fcf60f410089420a9499f08086. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:25.896031 13032 tablet_service.cc:1467] Processing CreateTablet for tablet 023517b9d4ee43f497d98c4d7a6037df (DEFAULT_TABLE table=rebalance_test_table_17 [id=52f6b247496e4c3eb63eef4f16160012]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:25.896253 13032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 023517b9d4ee43f497d98c4d7a6037df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:25.898106 15238 tablet_bootstrap.cc:492] T 023517b9d4ee43f497d98c4d7a6037df P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:25.898170 15236 tablet_bootstrap.cc:492] T 3bb148fcf60f410089420a9499f08086 P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:25.898169 13164 tablet_service.cc:1467] Processing CreateTablet for tablet 866723d26350414197ec27544277db9c (DEFAULT_TABLE table=rebalance_test_table_17 [id=52f6b247496e4c3eb63eef4f16160012]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:25.898375 13164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 866723d26350414197ec27544277db9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:25.898864 15236 tablet_bootstrap.cc:654] T 3bb148fcf60f410089420a9499f08086 P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:25.898952 15238 tablet_bootstrap.cc:654] T 023517b9d4ee43f497d98c4d7a6037df P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:25.899899 15238 tablet_bootstrap.cc:492] T 023517b9d4ee43f497d98c4d7a6037df P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:25.899915 15236 tablet_bootstrap.cc:492] T 3bb148fcf60f410089420a9499f08086 P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:25.899957 15238 ts_tablet_manager.cc:1367] T 023517b9d4ee43f497d98c4d7a6037df P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:25.899997 15236 ts_tablet_manager.cc:1367] T 3bb148fcf60f410089420a9499f08086 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:25.900301 15238 raft_consensus.cc:348] T 023517b9d4ee43f497d98c4d7a6037df P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:25.900379 15238 raft_consensus.cc:374] T 023517b9d4ee43f497d98c4d7a6037df P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:25.900398 15238 raft_consensus.cc:729] T 023517b9d4ee43f497d98c4d7a6037df P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:25.900430 13425 tablet_service.cc:1467] Processing CreateTablet for tablet 866723d26350414197ec27544277db9c (DEFAULT_TABLE table=rebalance_test_table_17 [id=52f6b247496e4c3eb63eef4f16160012]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:25.900430 15236 raft_consensus.cc:348] T 3bb148fcf60f410089420a9499f08086 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:25.900527 15238 consensus_queue.cc:260] T 023517b9d4ee43f497d98c4d7a6037df P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:25.900591 15236 raft_consensus.cc:374] T 3bb148fcf60f410089420a9499f08086 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:25.900620 15236 raft_consensus.cc:729] T 3bb148fcf60f410089420a9499f08086 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:25.900698 13425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 866723d26350414197ec27544277db9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:25.900763 15238 ts_tablet_manager.cc:1398] T 023517b9d4ee43f497d98c4d7a6037df P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:25.900735 15236 consensus_queue.cc:260] T 3bb148fcf60f410089420a9499f08086 P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:25.900997 15236 ts_tablet_manager.cc:1398] T 3bb148fcf60f410089420a9499f08086 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:25.902278 15236 tablet_bootstrap.cc:492] T 866723d26350414197ec27544277db9c P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:25.903123 15236 tablet_bootstrap.cc:654] T 866723d26350414197ec27544277db9c P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:25.903492 13296 tablet_service.cc:1467] Processing CreateTablet for tablet 3bb148fcf60f410089420a9499f08086 (DEFAULT_TABLE table=rebalance_test_table_17 [id=52f6b247496e4c3eb63eef4f16160012]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:25.903752 13296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bb148fcf60f410089420a9499f08086. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:25.906100 13295 tablet_service.cc:1467] Processing CreateTablet for tablet 866723d26350414197ec27544277db9c (DEFAULT_TABLE table=rebalance_test_table_17 [id=52f6b247496e4c3eb63eef4f16160012]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:25.906333 13295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 866723d26350414197ec27544277db9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:25.906419 15242 tablet_bootstrap.cc:492] T 3bb148fcf60f410089420a9499f08086 P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:25.907481 15242 tablet_bootstrap.cc:654] T 3bb148fcf60f410089420a9499f08086 P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:25.908279 13294 tablet_service.cc:1467] Processing CreateTablet for tablet 023517b9d4ee43f497d98c4d7a6037df (DEFAULT_TABLE table=rebalance_test_table_17 [id=52f6b247496e4c3eb63eef4f16160012]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:25.908511 13294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 023517b9d4ee43f497d98c4d7a6037df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:25.909162 15237 tablet_bootstrap.cc:492] T 3bb148fcf60f410089420a9499f08086 P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:25.909947 15237 tablet_bootstrap.cc:654] T 3bb148fcf60f410089420a9499f08086 P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:25.908591 15242 tablet_bootstrap.cc:492] T 3bb148fcf60f410089420a9499f08086 P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:25.910033 15242 ts_tablet_manager.cc:1367] T 3bb148fcf60f410089420a9499f08086 P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:56:25.910591 15242 raft_consensus.cc:348] T 3bb148fcf60f410089420a9499f08086 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:25.910648 13164 tablet_service.cc:1467] Processing CreateTablet for tablet 023517b9d4ee43f497d98c4d7a6037df (DEFAULT_TABLE table=rebalance_test_table_17 [id=52f6b247496e4c3eb63eef4f16160012]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:25.910775 15242 raft_consensus.cc:374] T 3bb148fcf60f410089420a9499f08086 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:25.910813 15242 raft_consensus.cc:729] T 3bb148fcf60f410089420a9499f08086 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:25.910883 13164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 023517b9d4ee43f497d98c4d7a6037df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:25.910954 15242 consensus_queue.cc:260] T 3bb148fcf60f410089420a9499f08086 P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:25.911288 15242 ts_tablet_manager.cc:1398] T 3bb148fcf60f410089420a9499f08086 P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:25.911429 15242 tablet_bootstrap.cc:492] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:25.911918 15236 tablet_bootstrap.cc:492] T 866723d26350414197ec27544277db9c P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:25.912001 15236 ts_tablet_manager.cc:1367] T 866723d26350414197ec27544277db9c P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 13:56:25.912423 15236 raft_consensus.cc:348] T 866723d26350414197ec27544277db9c P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:25.912549 15236 raft_consensus.cc:374] T 866723d26350414197ec27544277db9c P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:25.912580 15236 raft_consensus.cc:729] T 866723d26350414197ec27544277db9c P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:25.912693 15236 consensus_queue.cc:260] T 866723d26350414197ec27544277db9c P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:25.912988 15236 ts_tablet_manager.cc:1398] T 866723d26350414197ec27544277db9c P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:25.914701 15237 tablet_bootstrap.cc:492] T 3bb148fcf60f410089420a9499f08086 P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:25.914778 15237 ts_tablet_manager.cc:1367] T 3bb148fcf60f410089420a9499f08086 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 13:56:25.915158 15237 raft_consensus.cc:348] T 3bb148fcf60f410089420a9499f08086 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:25.915248 15237 raft_consensus.cc:374] T 3bb148fcf60f410089420a9499f08086 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:25.915277 15237 raft_consensus.cc:729] T 3bb148fcf60f410089420a9499f08086 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:25.915383 15237 consensus_queue.cc:260] T 3bb148fcf60f410089420a9499f08086 P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:25.915628 15237 ts_tablet_manager.cc:1398] T 3bb148fcf60f410089420a9499f08086 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:25.915756 15237 tablet_bootstrap.cc:492] T 866723d26350414197ec27544277db9c P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:25.916502 15242 tablet_bootstrap.cc:654] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:25.917609 15237 tablet_bootstrap.cc:654] T 866723d26350414197ec27544277db9c P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:25.917723 15242 tablet_bootstrap.cc:492] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:25.917802 15242 ts_tablet_manager.cc:1367] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 13:56:25.918219 15242 raft_consensus.cc:348] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:25.918329 15242 raft_consensus.cc:374] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:25.918359 15242 raft_consensus.cc:729] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:25.918480 15242 consensus_queue.cc:260] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:25.918709 15237 tablet_bootstrap.cc:492] T 866723d26350414197ec27544277db9c P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:25.918747 15242 ts_tablet_manager.cc:1398] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:25.918778 15237 ts_tablet_manager.cc:1367] T 866723d26350414197ec27544277db9c P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:56:25.919021 15242 tablet_bootstrap.cc:492] T 023517b9d4ee43f497d98c4d7a6037df P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:25.919163 15237 raft_consensus.cc:348] T 866723d26350414197ec27544277db9c P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:25.919247 15237 raft_consensus.cc:374] T 866723d26350414197ec27544277db9c P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:25.919275 15237 raft_consensus.cc:729] T 866723d26350414197ec27544277db9c P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:25.919377 15237 consensus_queue.cc:260] T 866723d26350414197ec27544277db9c P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:25.919631 15237 ts_tablet_manager.cc:1398] T 866723d26350414197ec27544277db9c P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:25.919777 15237 tablet_bootstrap.cc:492] T 023517b9d4ee43f497d98c4d7a6037df P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:25.919835 15242 tablet_bootstrap.cc:654] T 023517b9d4ee43f497d98c4d7a6037df P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:25.921336 15242 tablet_bootstrap.cc:492] T 023517b9d4ee43f497d98c4d7a6037df P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:25.921470 15242 ts_tablet_manager.cc:1367] T 023517b9d4ee43f497d98c4d7a6037df P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:56:25.921902 15242 raft_consensus.cc:348] T 023517b9d4ee43f497d98c4d7a6037df P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:25.922034 15237 tablet_bootstrap.cc:654] T 023517b9d4ee43f497d98c4d7a6037df P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:25.922076 15242 raft_consensus.cc:374] T 023517b9d4ee43f497d98c4d7a6037df P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:25.922153 15242 raft_consensus.cc:729] T 023517b9d4ee43f497d98c4d7a6037df P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:25.922444 15242 consensus_queue.cc:260] T 023517b9d4ee43f497d98c4d7a6037df P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:25.922834 15242 ts_tablet_manager.cc:1398] T 023517b9d4ee43f497d98c4d7a6037df P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:25.923900 15237 tablet_bootstrap.cc:492] T 023517b9d4ee43f497d98c4d7a6037df P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:25.923977 15237 ts_tablet_manager.cc:1367] T 023517b9d4ee43f497d98c4d7a6037df P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 13:56:25.925012 15237 raft_consensus.cc:348] T 023517b9d4ee43f497d98c4d7a6037df P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:25.925146 15237 raft_consensus.cc:374] T 023517b9d4ee43f497d98c4d7a6037df P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:25.925189 15237 raft_consensus.cc:729] T 023517b9d4ee43f497d98c4d7a6037df P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:25.925307 15237 consensus_queue.cc:260] T 023517b9d4ee43f497d98c4d7a6037df P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:25.925652 15237 ts_tablet_manager.cc:1398] T 023517b9d4ee43f497d98c4d7a6037df P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:25.985344 15152 raft_consensus.cc:482] T 3bb148fcf60f410089420a9499f08086 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:25.985497 15152 raft_consensus.cc:504] T 3bb148fcf60f410089420a9499f08086 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:25.985888 15152 leader_election.cc:288] T 3bb148fcf60f410089420a9499f08086 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:25.986248 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bb148fcf60f410089420a9499f08086" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
I20240503 13:56:25.986332 13184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bb148fcf60f410089420a9499f08086" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770" is_pre_election: true
I20240503 13:56:25.986469 13446 raft_consensus.cc:2454] T 3bb148fcf60f410089420a9499f08086 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 0.
I20240503 13:56:25.986498 13184 raft_consensus.cc:2454] T 3bb148fcf60f410089420a9499f08086 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 0.
I20240503 13:56:25.986814 13252 leader_election.cc:302] T 3bb148fcf60f410089420a9499f08086 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ccf5a4ac450414c9d3579bd5800b770, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:25.987157 15152 raft_consensus.cc:2781] T 3bb148fcf60f410089420a9499f08086 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:25.987227 15152 raft_consensus.cc:482] T 3bb148fcf60f410089420a9499f08086 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:25.987264 15152 raft_consensus.cc:3037] T 3bb148fcf60f410089420a9499f08086 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:25.988112 15152 raft_consensus.cc:504] T 3bb148fcf60f410089420a9499f08086 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:25.988435 15152 leader_election.cc:288] T 3bb148fcf60f410089420a9499f08086 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: Requested vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:25.988749 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bb148fcf60f410089420a9499f08086" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
I20240503 13:56:25.988817 13184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bb148fcf60f410089420a9499f08086" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770"
I20240503 13:56:25.988891 13446 raft_consensus.cc:3037] T 3bb148fcf60f410089420a9499f08086 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:25.988922 13184 raft_consensus.cc:3037] T 3bb148fcf60f410089420a9499f08086 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:25.989619 13184 raft_consensus.cc:2454] T 3bb148fcf60f410089420a9499f08086 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 1.
I20240503 13:56:25.989630 13446 raft_consensus.cc:2454] T 3bb148fcf60f410089420a9499f08086 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 1.
I20240503 13:56:25.989900 13252 leader_election.cc:302] T 3bb148fcf60f410089420a9499f08086 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ccf5a4ac450414c9d3579bd5800b770, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:25.990085 15152 raft_consensus.cc:2781] T 3bb148fcf60f410089420a9499f08086 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:25.990165 15152 raft_consensus.cc:686] T 3bb148fcf60f410089420a9499f08086 P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Becoming Leader. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Running, Role: LEADER
I20240503 13:56:25.990291 15152 consensus_queue.cc:237] T 3bb148fcf60f410089420a9499f08086 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:25.992192 12909 catalog_manager.cc:5484] T 3bb148fcf60f410089420a9499f08086 P ba0194098aa84449ae17e8a0d4c43703 reported cstate change: term changed from 0 to 1, leader changed from <none> to ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67). New cstate: current_term: 1 leader_uuid: "ba0194098aa84449ae17e8a0d4c43703" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:26.042578 15152 raft_consensus.cc:482] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:26.042719 15152 raft_consensus.cc:504] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:26.043274 15152 leader_election.cc:288] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387), 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:26.043910 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "866723d26350414197ec27544277db9c" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
I20240503 13:56:26.043941 13184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "866723d26350414197ec27544277db9c" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770" is_pre_election: true
I20240503 13:56:26.044071 13446 raft_consensus.cc:2454] T 866723d26350414197ec27544277db9c P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 0.
I20240503 13:56:26.044071 13184 raft_consensus.cc:2454] T 866723d26350414197ec27544277db9c P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 0.
I20240503 13:56:26.044391 13249 leader_election.cc:302] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:26.044610 15152 raft_consensus.cc:2781] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:26.044685 15152 raft_consensus.cc:482] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:26.044724 15152 raft_consensus.cc:3037] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:26.045562 15152 raft_consensus.cc:504] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:26.045912 15152 leader_election.cc:288] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: Requested vote from peers 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387), 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:26.046240 13184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "866723d26350414197ec27544277db9c" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770"
I20240503 13:56:26.046247 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "866723d26350414197ec27544277db9c" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
I20240503 13:56:26.046353 13184 raft_consensus.cc:3037] T 866723d26350414197ec27544277db9c P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:26.046370 13446 raft_consensus.cc:3037] T 866723d26350414197ec27544277db9c P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:26.047173 13446 raft_consensus.cc:2454] T 866723d26350414197ec27544277db9c P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 1.
I20240503 13:56:26.047210 13184 raft_consensus.cc:2454] T 866723d26350414197ec27544277db9c P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 1.
I20240503 13:56:26.047472 13252 leader_election.cc:302] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ccf5a4ac450414c9d3579bd5800b770, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:26.047788 15152 raft_consensus.cc:2781] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:26.047868 15152 raft_consensus.cc:686] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Becoming Leader. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Running, Role: LEADER
I20240503 13:56:26.048009 15152 consensus_queue.cc:237] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:26.049968 12909 catalog_manager.cc:5484] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 reported cstate change: term changed from 0 to 1, leader changed from <none> to ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67). New cstate: current_term: 1 leader_uuid: "ba0194098aa84449ae17e8a0d4c43703" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } }
I20240503 13:56:26.060052 14984 raft_consensus.cc:482] T 023517b9d4ee43f497d98c4d7a6037df P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:26.060163 14984 raft_consensus.cc:504] T 023517b9d4ee43f497d98c4d7a6037df P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:26.060573 14984 leader_election.cc:288] T 023517b9d4ee43f497d98c4d7a6037df P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387), ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:26.060957 13184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "023517b9d4ee43f497d98c4d7a6037df" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770" is_pre_election: true
I20240503 13:56:26.060979 13314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "023517b9d4ee43f497d98c4d7a6037df" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703" is_pre_election: true
I20240503 13:56:26.061107 13184 raft_consensus.cc:2454] T 023517b9d4ee43f497d98c4d7a6037df P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 0.
I20240503 13:56:26.061115 13314 raft_consensus.cc:2454] T 023517b9d4ee43f497d98c4d7a6037df P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 0.
I20240503 13:56:26.061398 12990 leader_election.cc:302] T 023517b9d4ee43f497d98c4d7a6037df P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 8ccf5a4ac450414c9d3579bd5800b770; no voters:
I20240503 13:56:26.061571 14984 raft_consensus.cc:2781] T 023517b9d4ee43f497d98c4d7a6037df P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:26.061633 14984 raft_consensus.cc:482] T 023517b9d4ee43f497d98c4d7a6037df P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:26.061667 14984 raft_consensus.cc:3037] T 023517b9d4ee43f497d98c4d7a6037df P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:26.062631 14984 raft_consensus.cc:504] T 023517b9d4ee43f497d98c4d7a6037df P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:26.062920 14984 leader_election.cc:288] T 023517b9d4ee43f497d98c4d7a6037df P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 election: Requested vote from peers 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387), ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:26.063213 13184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "023517b9d4ee43f497d98c4d7a6037df" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770"
I20240503 13:56:26.063316 13184 raft_consensus.cc:3037] T 023517b9d4ee43f497d98c4d7a6037df P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:26.063289 13314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "023517b9d4ee43f497d98c4d7a6037df" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703"
I20240503 13:56:26.063392 13314 raft_consensus.cc:3037] T 023517b9d4ee43f497d98c4d7a6037df P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:26.064044 13184 raft_consensus.cc:2454] T 023517b9d4ee43f497d98c4d7a6037df P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 1.
I20240503 13:56:26.064148 13314 raft_consensus.cc:2454] T 023517b9d4ee43f497d98c4d7a6037df P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 1.
I20240503 13:56:26.064311 12990 leader_election.cc:302] T 023517b9d4ee43f497d98c4d7a6037df P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 8ccf5a4ac450414c9d3579bd5800b770; no voters:
I20240503 13:56:26.064450 14984 raft_consensus.cc:2781] T 023517b9d4ee43f497d98c4d7a6037df P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:26.064597 14984 raft_consensus.cc:686] T 023517b9d4ee43f497d98c4d7a6037df P 60ba3ffe3b7746469834cd91c84be14e [term 1 LEADER]: Becoming Leader. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Running, Role: LEADER
I20240503 13:56:26.064745 14984 consensus_queue.cc:237] T 023517b9d4ee43f497d98c4d7a6037df P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:26.066470 12909 catalog_manager.cc:5484] T 023517b9d4ee43f497d98c4d7a6037df P 60ba3ffe3b7746469834cd91c84be14e reported cstate change: term changed from 0 to 1, leader changed from <none> to 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65). New cstate: current_term: 1 leader_uuid: "60ba3ffe3b7746469834cd91c84be14e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
W20240503 13:56:26.211591 15252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:56:26.211849 15252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:56:26.215265 15252 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_17'
I20240503 13:56:26.304757 13447 raft_consensus.cc:1261] T 3bb148fcf60f410089420a9499f08086 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:26.305027 13184 raft_consensus.cc:1261] T 3bb148fcf60f410089420a9499f08086 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:26.305572 15152 consensus_queue.cc:1035] T 3bb148fcf60f410089420a9499f08086 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:26.306108 13184 raft_consensus.cc:1261] T 866723d26350414197ec27544277db9c P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:26.306530 15206 consensus_queue.cc:1035] T 3bb148fcf60f410089420a9499f08086 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:26.307121 15206 consensus_queue.cc:1035] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:26.309536 13447 raft_consensus.cc:1261] T 866723d26350414197ec27544277db9c P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:26.310061 15152 consensus_queue.cc:1035] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:26.318307 13183 raft_consensus.cc:1261] T 023517b9d4ee43f497d98c4d7a6037df P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Refusing update from remote peer 60ba3ffe3b7746469834cd91c84be14e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:26.319569 14984 consensus_queue.cc:1035] T 023517b9d4ee43f497d98c4d7a6037df P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:26.321398 13314 raft_consensus.cc:1261] T 023517b9d4ee43f497d98c4d7a6037df P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Refusing update from remote peer 60ba3ffe3b7746469834cd91c84be14e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:26.325163 15215 mvcc.cc:205] Tried to move back new op lower bound from 7023593825499160576 to 7023593824217518080. Current Snapshot: MvccSnapshot[applied={T|T < 7023593825499160576}]
I20240503 13:56:26.328689 14984 consensus_queue.cc:1035] T 023517b9d4ee43f497d98c4d7a6037df P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:26.358559 15208 mvcc.cc:205] Tried to move back new op lower bound from 7023593825500839936 to 7023593824453734400. Current Snapshot: MvccSnapshot[applied={T|T < 7023593825500839936}]
INSERT report
rows total: 2000
time total: 121.211 ms
time per row: 0.0606057 ms
I20240503 13:56:26.381804 12903 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalance_test_table_18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:56:26.388039 13425 tablet_service.cc:1467] Processing CreateTablet for tablet 760e11f3264f4bf5b1865f1c2c46cd59 (DEFAULT_TABLE table=rebalance_test_table_18 [id=aa180ed819e24d89be0f5570e50d3c82]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:26.388401 13425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 760e11f3264f4bf5b1865f1c2c46cd59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:26.388475 13032 tablet_service.cc:1467] Processing CreateTablet for tablet 760e11f3264f4bf5b1865f1c2c46cd59 (DEFAULT_TABLE table=rebalance_test_table_18 [id=aa180ed819e24d89be0f5570e50d3c82]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:26.388655 13427 tablet_service.cc:1467] Processing CreateTablet for tablet 4bd42569fd3b4b39b0bd489acebbad10 (DEFAULT_TABLE table=rebalance_test_table_18 [id=aa180ed819e24d89be0f5570e50d3c82]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:26.388774 13032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 760e11f3264f4bf5b1865f1c2c46cd59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:26.388860 13427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bd42569fd3b4b39b0bd489acebbad10. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:26.388039 13426 tablet_service.cc:1467] Processing CreateTablet for tablet 81d47f705c0b4128a84e4bf7d274a659 (DEFAULT_TABLE table=rebalance_test_table_18 [id=aa180ed819e24d89be0f5570e50d3c82]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:26.390643 15238 tablet_bootstrap.cc:492] T 760e11f3264f4bf5b1865f1c2c46cd59 P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:26.390792 13426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81d47f705c0b4128a84e4bf7d274a659. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:26.391407 13165 tablet_service.cc:1467] Processing CreateTablet for tablet 4bd42569fd3b4b39b0bd489acebbad10 (DEFAULT_TABLE table=rebalance_test_table_18 [id=aa180ed819e24d89be0f5570e50d3c82]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:26.391609 15238 tablet_bootstrap.cc:654] T 760e11f3264f4bf5b1865f1c2c46cd59 P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:26.391688 13165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bd42569fd3b4b39b0bd489acebbad10. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:26.391837 13294 tablet_service.cc:1467] Processing CreateTablet for tablet 81d47f705c0b4128a84e4bf7d274a659 (DEFAULT_TABLE table=rebalance_test_table_18 [id=aa180ed819e24d89be0f5570e50d3c82]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:26.392097 13294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81d47f705c0b4128a84e4bf7d274a659. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:26.390808 15236 tablet_bootstrap.cc:492] T 4bd42569fd3b4b39b0bd489acebbad10 P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:26.391407 13164 tablet_service.cc:1467] Processing CreateTablet for tablet 760e11f3264f4bf5b1865f1c2c46cd59 (DEFAULT_TABLE table=rebalance_test_table_18 [id=aa180ed819e24d89be0f5570e50d3c82]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:26.392762 13164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 760e11f3264f4bf5b1865f1c2c46cd59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:26.393364 15236 tablet_bootstrap.cc:654] T 4bd42569fd3b4b39b0bd489acebbad10 P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:26.393553 15237 tablet_bootstrap.cc:492] T 4bd42569fd3b4b39b0bd489acebbad10 P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:26.394325 15237 tablet_bootstrap.cc:654] T 4bd42569fd3b4b39b0bd489acebbad10 P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:26.394763 13032 tablet_service.cc:1467] Processing CreateTablet for tablet 81d47f705c0b4128a84e4bf7d274a659 (DEFAULT_TABLE table=rebalance_test_table_18 [id=aa180ed819e24d89be0f5570e50d3c82]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:26.394784 13033 tablet_service.cc:1467] Processing CreateTablet for tablet 4bd42569fd3b4b39b0bd489acebbad10 (DEFAULT_TABLE table=rebalance_test_table_18 [id=aa180ed819e24d89be0f5570e50d3c82]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:26.395013 13032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81d47f705c0b4128a84e4bf7d274a659. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:26.396323 15242 tablet_bootstrap.cc:492] T 81d47f705c0b4128a84e4bf7d274a659 P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:26.396684 13033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bd42569fd3b4b39b0bd489acebbad10. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:26.397248 15242 tablet_bootstrap.cc:654] T 81d47f705c0b4128a84e4bf7d274a659 P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:26.397641 15237 tablet_bootstrap.cc:492] T 4bd42569fd3b4b39b0bd489acebbad10 P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:26.397657 15236 tablet_bootstrap.cc:492] T 4bd42569fd3b4b39b0bd489acebbad10 P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:26.397728 15237 ts_tablet_manager.cc:1367] T 4bd42569fd3b4b39b0bd489acebbad10 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 13:56:26.397742 15236 ts_tablet_manager.cc:1367] T 4bd42569fd3b4b39b0bd489acebbad10 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 13:56:26.398171 15236 raft_consensus.cc:348] T 4bd42569fd3b4b39b0bd489acebbad10 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:26.398185 15237 raft_consensus.cc:348] T 4bd42569fd3b4b39b0bd489acebbad10 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:26.398293 15236 raft_consensus.cc:374] T 4bd42569fd3b4b39b0bd489acebbad10 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:26.398293 15237 raft_consensus.cc:374] T 4bd42569fd3b4b39b0bd489acebbad10 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:26.398326 15237 raft_consensus.cc:729] T 4bd42569fd3b4b39b0bd489acebbad10 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:26.398326 15236 raft_consensus.cc:729] T 4bd42569fd3b4b39b0bd489acebbad10 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:26.398475 15238 tablet_bootstrap.cc:492] T 760e11f3264f4bf5b1865f1c2c46cd59 P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:26.398449 15236 consensus_queue.cc:260] T 4bd42569fd3b4b39b0bd489acebbad10 P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:26.398555 15238 ts_tablet_manager.cc:1367] T 760e11f3264f4bf5b1865f1c2c46cd59 P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 13:56:26.398525 15237 consensus_queue.cc:260] T 4bd42569fd3b4b39b0bd489acebbad10 P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:26.398732 15236 ts_tablet_manager.cc:1398] T 4bd42569fd3b4b39b0bd489acebbad10 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:26.398823 15237 ts_tablet_manager.cc:1398] T 4bd42569fd3b4b39b0bd489acebbad10 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:26.398891 15236 tablet_bootstrap.cc:492] T 760e11f3264f4bf5b1865f1c2c46cd59 P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:26.398942 15238 raft_consensus.cc:348] T 760e11f3264f4bf5b1865f1c2c46cd59 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:26.399031 15237 tablet_bootstrap.cc:492] T 760e11f3264f4bf5b1865f1c2c46cd59 P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:26.399041 15238 raft_consensus.cc:374] T 760e11f3264f4bf5b1865f1c2c46cd59 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:26.399070 15238 raft_consensus.cc:729] T 760e11f3264f4bf5b1865f1c2c46cd59 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:26.399161 15238 consensus_queue.cc:260] T 760e11f3264f4bf5b1865f1c2c46cd59 P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:26.399422 15238 ts_tablet_manager.cc:1398] T 760e11f3264f4bf5b1865f1c2c46cd59 P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:26.399580 15238 tablet_bootstrap.cc:492] T 4bd42569fd3b4b39b0bd489acebbad10 P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:26.400296 15237 tablet_bootstrap.cc:654] T 760e11f3264f4bf5b1865f1c2c46cd59 P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:26.400381 15238 tablet_bootstrap.cc:654] T 4bd42569fd3b4b39b0bd489acebbad10 P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:26.401904 15238 tablet_bootstrap.cc:492] T 4bd42569fd3b4b39b0bd489acebbad10 P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:26.401997 15238 ts_tablet_manager.cc:1367] T 4bd42569fd3b4b39b0bd489acebbad10 P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 13:56:26.402405 15236 tablet_bootstrap.cc:654] T 760e11f3264f4bf5b1865f1c2c46cd59 P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:26.402529 15238 raft_consensus.cc:348] T 4bd42569fd3b4b39b0bd489acebbad10 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:26.402652 15238 raft_consensus.cc:374] T 4bd42569fd3b4b39b0bd489acebbad10 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:26.402691 15238 raft_consensus.cc:729] T 4bd42569fd3b4b39b0bd489acebbad10 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:26.402807 15238 consensus_queue.cc:260] T 4bd42569fd3b4b39b0bd489acebbad10 P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:26.403077 15237 tablet_bootstrap.cc:492] T 760e11f3264f4bf5b1865f1c2c46cd59 P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:26.403100 15238 ts_tablet_manager.cc:1398] T 4bd42569fd3b4b39b0bd489acebbad10 P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:26.403152 15237 ts_tablet_manager.cc:1367] T 760e11f3264f4bf5b1865f1c2c46cd59 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240503 13:56:26.403267 15238 tablet_bootstrap.cc:492] T 81d47f705c0b4128a84e4bf7d274a659 P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:26.403571 15242 tablet_bootstrap.cc:492] T 81d47f705c0b4128a84e4bf7d274a659 P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:26.403589 15237 raft_consensus.cc:348] T 760e11f3264f4bf5b1865f1c2c46cd59 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:26.403661 15242 ts_tablet_manager.cc:1367] T 81d47f705c0b4128a84e4bf7d274a659 P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240503 13:56:26.403688 15237 raft_consensus.cc:374] T 760e11f3264f4bf5b1865f1c2c46cd59 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:26.403715 15237 raft_consensus.cc:729] T 760e11f3264f4bf5b1865f1c2c46cd59 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:26.403862 15237 consensus_queue.cc:260] T 760e11f3264f4bf5b1865f1c2c46cd59 P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:26.404132 15238 tablet_bootstrap.cc:654] T 81d47f705c0b4128a84e4bf7d274a659 P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:26.404136 15242 raft_consensus.cc:348] T 81d47f705c0b4128a84e4bf7d274a659 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:26.404237 15242 raft_consensus.cc:374] T 81d47f705c0b4128a84e4bf7d274a659 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:26.404269 15242 raft_consensus.cc:729] T 81d47f705c0b4128a84e4bf7d274a659 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:26.404393 15242 consensus_queue.cc:260] T 81d47f705c0b4128a84e4bf7d274a659 P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:26.404735 15242 ts_tablet_manager.cc:1398] T 81d47f705c0b4128a84e4bf7d274a659 P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:26.404735 15237 ts_tablet_manager.cc:1398] T 760e11f3264f4bf5b1865f1c2c46cd59 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 13:56:26.404950 15096 raft_consensus.cc:482] T 4bd42569fd3b4b39b0bd489acebbad10 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:26.405033 15096 raft_consensus.cc:504] T 4bd42569fd3b4b39b0bd489acebbad10 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:26.405392 15096 leader_election.cc:288] T 4bd42569fd3b4b39b0bd489acebbad10 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387), 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:26.405828 13184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd42569fd3b4b39b0bd489acebbad10" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770" is_pre_election: true
I20240503 13:56:26.405972 13184 raft_consensus.cc:2454] T 4bd42569fd3b4b39b0bd489acebbad10 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 0.
I20240503 13:56:26.406385 12990 leader_election.cc:302] T 4bd42569fd3b4b39b0bd489acebbad10 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 8ccf5a4ac450414c9d3579bd5800b770; no voters:
I20240503 13:56:26.406746 15238 tablet_bootstrap.cc:492] T 81d47f705c0b4128a84e4bf7d274a659 P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:26.406834 15238 ts_tablet_manager.cc:1367] T 81d47f705c0b4128a84e4bf7d274a659 P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 13:56:26.406781 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd42569fd3b4b39b0bd489acebbad10" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
I20240503 13:56:26.406912 13447 raft_consensus.cc:2454] T 4bd42569fd3b4b39b0bd489acebbad10 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 0.
I20240503 13:56:26.407287 15238 raft_consensus.cc:348] T 81d47f705c0b4128a84e4bf7d274a659 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:26.407397 15238 raft_consensus.cc:374] T 81d47f705c0b4128a84e4bf7d274a659 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:26.407438 15238 raft_consensus.cc:729] T 81d47f705c0b4128a84e4bf7d274a659 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:26.407609 15238 consensus_queue.cc:260] T 81d47f705c0b4128a84e4bf7d274a659 P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:26.407828 14984 raft_consensus.cc:2781] T 4bd42569fd3b4b39b0bd489acebbad10 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:26.407881 14984 raft_consensus.cc:482] T 4bd42569fd3b4b39b0bd489acebbad10 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:26.407914 14984 raft_consensus.cc:3037] T 4bd42569fd3b4b39b0bd489acebbad10 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:26.407922 15238 ts_tablet_manager.cc:1398] T 81d47f705c0b4128a84e4bf7d274a659 P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:26.409054 14984 raft_consensus.cc:504] T 4bd42569fd3b4b39b0bd489acebbad10 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:26.409412 14984 leader_election.cc:288] T 4bd42569fd3b4b39b0bd489acebbad10 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 election: Requested vote from peers 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387), 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:26.409734 13184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd42569fd3b4b39b0bd489acebbad10" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770"
I20240503 13:56:26.409843 13184 raft_consensus.cc:3037] T 4bd42569fd3b4b39b0bd489acebbad10 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:26.410622 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd42569fd3b4b39b0bd489acebbad10" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
I20240503 13:56:26.410735 13447 raft_consensus.cc:3037] T 4bd42569fd3b4b39b0bd489acebbad10 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:26.410786 13184 raft_consensus.cc:2454] T 4bd42569fd3b4b39b0bd489acebbad10 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 1.
I20240503 13:56:26.411051 12990 leader_election.cc:302] T 4bd42569fd3b4b39b0bd489acebbad10 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 8ccf5a4ac450414c9d3579bd5800b770; no voters:
I20240503 13:56:26.411206 14984 raft_consensus.cc:2781] T 4bd42569fd3b4b39b0bd489acebbad10 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:26.411274 14984 raft_consensus.cc:686] T 4bd42569fd3b4b39b0bd489acebbad10 P 60ba3ffe3b7746469834cd91c84be14e [term 1 LEADER]: Becoming Leader. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Running, Role: LEADER
I20240503 13:56:26.411404 14984 consensus_queue.cc:237] T 4bd42569fd3b4b39b0bd489acebbad10 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:26.411693 13447 raft_consensus.cc:2454] T 4bd42569fd3b4b39b0bd489acebbad10 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 1.
I20240503 13:56:26.414323 12911 catalog_manager.cc:5484] T 4bd42569fd3b4b39b0bd489acebbad10 P 60ba3ffe3b7746469834cd91c84be14e reported cstate change: term changed from 0 to 1, leader changed from <none> to 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65). New cstate: current_term: 1 leader_uuid: "60ba3ffe3b7746469834cd91c84be14e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } }
I20240503 13:56:26.414635 15236 tablet_bootstrap.cc:492] T 760e11f3264f4bf5b1865f1c2c46cd59 P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:26.414728 15236 ts_tablet_manager.cc:1367] T 760e11f3264f4bf5b1865f1c2c46cd59 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240503 13:56:26.415169 15236 raft_consensus.cc:348] T 760e11f3264f4bf5b1865f1c2c46cd59 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:26.415282 15236 raft_consensus.cc:374] T 760e11f3264f4bf5b1865f1c2c46cd59 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:26.415315 15236 raft_consensus.cc:729] T 760e11f3264f4bf5b1865f1c2c46cd59 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:26.415439 15236 consensus_queue.cc:260] T 760e11f3264f4bf5b1865f1c2c46cd59 P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:26.415817 15236 ts_tablet_manager.cc:1398] T 760e11f3264f4bf5b1865f1c2c46cd59 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:26.416146 15236 tablet_bootstrap.cc:492] T 81d47f705c0b4128a84e4bf7d274a659 P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:26.417384 15236 tablet_bootstrap.cc:654] T 81d47f705c0b4128a84e4bf7d274a659 P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:26.418953 15236 tablet_bootstrap.cc:492] T 81d47f705c0b4128a84e4bf7d274a659 P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:26.420198 15236 ts_tablet_manager.cc:1367] T 81d47f705c0b4128a84e4bf7d274a659 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:56:26.420675 15236 raft_consensus.cc:348] T 81d47f705c0b4128a84e4bf7d274a659 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:26.420775 15236 raft_consensus.cc:374] T 81d47f705c0b4128a84e4bf7d274a659 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:26.420804 15236 raft_consensus.cc:729] T 81d47f705c0b4128a84e4bf7d274a659 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:26.420940 15236 consensus_queue.cc:260] T 81d47f705c0b4128a84e4bf7d274a659 P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:26.421207 15236 ts_tablet_manager.cc:1398] T 81d47f705c0b4128a84e4bf7d274a659 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:26.508590 15220 raft_consensus.cc:482] T 760e11f3264f4bf5b1865f1c2c46cd59 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:26.508713 15220 raft_consensus.cc:504] T 760e11f3264f4bf5b1865f1c2c46cd59 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:26.509076 15220 leader_election.cc:288] T 760e11f3264f4bf5b1865f1c2c46cd59 P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695), 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:26.509446 13053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "760e11f3264f4bf5b1865f1c2c46cd59" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e" is_pre_election: true
I20240503 13:56:26.509454 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "760e11f3264f4bf5b1865f1c2c46cd59" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
I20240503 13:56:26.509584 13053 raft_consensus.cc:2454] T 760e11f3264f4bf5b1865f1c2c46cd59 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 0.
I20240503 13:56:26.509616 13447 raft_consensus.cc:2454] T 760e11f3264f4bf5b1865f1c2c46cd59 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 0.
I20240503 13:56:26.509898 13119 leader_election.cc:302] T 760e11f3264f4bf5b1865f1c2c46cd59 P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 8ccf5a4ac450414c9d3579bd5800b770; no voters:
I20240503 13:56:26.510061 15220 raft_consensus.cc:2781] T 760e11f3264f4bf5b1865f1c2c46cd59 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:26.510123 15220 raft_consensus.cc:482] T 760e11f3264f4bf5b1865f1c2c46cd59 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:26.510155 15220 raft_consensus.cc:3037] T 760e11f3264f4bf5b1865f1c2c46cd59 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:26.510998 15220 raft_consensus.cc:504] T 760e11f3264f4bf5b1865f1c2c46cd59 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:26.511322 15220 leader_election.cc:288] T 760e11f3264f4bf5b1865f1c2c46cd59 P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 election: Requested vote from peers 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695), 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:26.511651 13053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "760e11f3264f4bf5b1865f1c2c46cd59" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e"
I20240503 13:56:26.511695 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "760e11f3264f4bf5b1865f1c2c46cd59" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
I20240503 13:56:26.511773 13053 raft_consensus.cc:3037] T 760e11f3264f4bf5b1865f1c2c46cd59 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:26.511794 13447 raft_consensus.cc:3037] T 760e11f3264f4bf5b1865f1c2c46cd59 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:26.512720 13447 raft_consensus.cc:2454] T 760e11f3264f4bf5b1865f1c2c46cd59 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 1.
I20240503 13:56:26.512720 13053 raft_consensus.cc:2454] T 760e11f3264f4bf5b1865f1c2c46cd59 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 1.
I20240503 13:56:26.512984 13118 leader_election.cc:302] T 760e11f3264f4bf5b1865f1c2c46cd59 P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, 8ccf5a4ac450414c9d3579bd5800b770; no voters:
I20240503 13:56:26.513185 15220 raft_consensus.cc:2781] T 760e11f3264f4bf5b1865f1c2c46cd59 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:26.513257 15220 raft_consensus.cc:686] T 760e11f3264f4bf5b1865f1c2c46cd59 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 LEADER]: Becoming Leader. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Running, Role: LEADER
I20240503 13:56:26.513394 15220 consensus_queue.cc:237] T 760e11f3264f4bf5b1865f1c2c46cd59 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:26.515276 12905 catalog_manager.cc:5484] T 760e11f3264f4bf5b1865f1c2c46cd59 P 8ccf5a4ac450414c9d3579bd5800b770 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66). New cstate: current_term: 1 leader_uuid: "8ccf5a4ac450414c9d3579bd5800b770" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } }
I20240503 13:56:26.565378 14911 raft_consensus.cc:482] T 81d47f705c0b4128a84e4bf7d274a659 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:26.565510 14911 raft_consensus.cc:504] T 81d47f705c0b4128a84e4bf7d274a659 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:26.565891 14911 leader_election.cc:288] T 81d47f705c0b4128a84e4bf7d274a659 P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695), ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:26.566256 13053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d47f705c0b4128a84e4bf7d274a659" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e" is_pre_election: true
I20240503 13:56:26.566373 13053 raft_consensus.cc:2454] T 81d47f705c0b4128a84e4bf7d274a659 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 0.
I20240503 13:56:26.566321 13314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d47f705c0b4128a84e4bf7d274a659" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703" is_pre_election: true
I20240503 13:56:26.566473 13314 raft_consensus.cc:2454] T 81d47f705c0b4128a84e4bf7d274a659 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 0.
I20240503 13:56:26.566668 13381 leader_election.cc:302] T 81d47f705c0b4128a84e4bf7d274a659 P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 7690ee79577c45e3a2c8f8b552c0bff8; no voters:
I20240503 13:56:26.566885 14911 raft_consensus.cc:2781] T 81d47f705c0b4128a84e4bf7d274a659 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:26.566942 14911 raft_consensus.cc:482] T 81d47f705c0b4128a84e4bf7d274a659 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:26.566985 14911 raft_consensus.cc:3037] T 81d47f705c0b4128a84e4bf7d274a659 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:26.568202 14911 raft_consensus.cc:504] T 81d47f705c0b4128a84e4bf7d274a659 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:26.568742 14911 leader_election.cc:288] T 81d47f705c0b4128a84e4bf7d274a659 P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 election: Requested vote from peers 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695), ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:26.569106 13314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d47f705c0b4128a84e4bf7d274a659" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703"
I20240503 13:56:26.569121 13053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d47f705c0b4128a84e4bf7d274a659" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e"
I20240503 13:56:26.569243 13053 raft_consensus.cc:3037] T 81d47f705c0b4128a84e4bf7d274a659 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:26.569240 13314 raft_consensus.cc:3037] T 81d47f705c0b4128a84e4bf7d274a659 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:26.570031 13314 raft_consensus.cc:2454] T 81d47f705c0b4128a84e4bf7d274a659 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 1.
I20240503 13:56:26.570227 13053 raft_consensus.cc:2454] T 81d47f705c0b4128a84e4bf7d274a659 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 1.
I20240503 13:56:26.570428 13383 leader_election.cc:302] T 81d47f705c0b4128a84e4bf7d274a659 P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:26.570595 14911 raft_consensus.cc:2781] T 81d47f705c0b4128a84e4bf7d274a659 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:26.570690 14911 raft_consensus.cc:686] T 81d47f705c0b4128a84e4bf7d274a659 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 LEADER]: Becoming Leader. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Running, Role: LEADER
I20240503 13:56:26.570828 14911 consensus_queue.cc:237] T 81d47f705c0b4128a84e4bf7d274a659 P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:26.573047 12905 catalog_manager.cc:5484] T 81d47f705c0b4128a84e4bf7d274a659 P 7690ee79577c45e3a2c8f8b552c0bff8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68). New cstate: current_term: 1 leader_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
W20240503 13:56:26.709828 15297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:56:26.710093 15297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:56:26.713272 15297 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_18'
I20240503 13:56:26.800257 15096 consensus_queue.cc:1035] T 4bd42569fd3b4b39b0bd489acebbad10 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:26.805595 13053 raft_consensus.cc:1261] T 81d47f705c0b4128a84e4bf7d274a659 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Refusing update from remote peer 7690ee79577c45e3a2c8f8b552c0bff8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:26.806282 14911 consensus_queue.cc:1035] T 81d47f705c0b4128a84e4bf7d274a659 P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:26.809345 13316 raft_consensus.cc:1261] T 81d47f705c0b4128a84e4bf7d274a659 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Refusing update from remote peer 7690ee79577c45e3a2c8f8b552c0bff8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:26.816521 15295 consensus_queue.cc:1035] T 81d47f705c0b4128a84e4bf7d274a659 P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:26.821951 14984 consensus_queue.cc:1035] T 4bd42569fd3b4b39b0bd489acebbad10 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:26.838505 13446 raft_consensus.cc:1261] T 760e11f3264f4bf5b1865f1c2c46cd59 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer 8ccf5a4ac450414c9d3579bd5800b770: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:26.839028 15220 consensus_queue.cc:1035] T 760e11f3264f4bf5b1865f1c2c46cd59 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:26.851003 13053 raft_consensus.cc:1261] T 760e11f3264f4bf5b1865f1c2c46cd59 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Refusing update from remote peer 8ccf5a4ac450414c9d3579bd5800b770: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:26.855679 14572 consensus_queue.cc:1035] T 760e11f3264f4bf5b1865f1c2c46cd59 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 117.71 ms
time per row: 0.0588549 ms
I20240503 13:56:26.887025 12906 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalance_test_table_19"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:56:26.893381 13032 tablet_service.cc:1467] Processing CreateTablet for tablet dc544319be0b484f8e9e5a3fbc40f530 (DEFAULT_TABLE table=rebalance_test_table_19 [id=e6609c16048348029c01374cb5a68ce8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:26.893741 13032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc544319be0b484f8e9e5a3fbc40f530. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:26.893788 13426 tablet_service.cc:1467] Processing CreateTablet for tablet fd6a3d908840456f82c346c4bf1d8160 (DEFAULT_TABLE table=rebalance_test_table_19 [id=e6609c16048348029c01374cb5a68ce8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:26.893800 13425 tablet_service.cc:1467] Processing CreateTablet for tablet 84f1ab11e74045f8a9d7033c5ead0c14 (DEFAULT_TABLE table=rebalance_test_table_19 [id=e6609c16048348029c01374cb5a68ce8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:26.894049 13426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd6a3d908840456f82c346c4bf1d8160. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:26.894194 13425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84f1ab11e74045f8a9d7033c5ead0c14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:26.895315 13164 tablet_service.cc:1467] Processing CreateTablet for tablet dc544319be0b484f8e9e5a3fbc40f530 (DEFAULT_TABLE table=rebalance_test_table_19 [id=e6609c16048348029c01374cb5a68ce8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:26.895357 13165 tablet_service.cc:1467] Processing CreateTablet for tablet 84f1ab11e74045f8a9d7033c5ead0c14 (DEFAULT_TABLE table=rebalance_test_table_19 [id=e6609c16048348029c01374cb5a68ce8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:26.895596 13164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc544319be0b484f8e9e5a3fbc40f530. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:26.896036 13294 tablet_service.cc:1467] Processing CreateTablet for tablet dc544319be0b484f8e9e5a3fbc40f530 (DEFAULT_TABLE table=rebalance_test_table_19 [id=e6609c16048348029c01374cb5a68ce8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:26.896308 13294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc544319be0b484f8e9e5a3fbc40f530. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:26.896389 13033 tablet_service.cc:1467] Processing CreateTablet for tablet fd6a3d908840456f82c346c4bf1d8160 (DEFAULT_TABLE table=rebalance_test_table_19 [id=e6609c16048348029c01374cb5a68ce8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:26.896690 13033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd6a3d908840456f82c346c4bf1d8160. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:26.897601 15236 tablet_bootstrap.cc:492] T 84f1ab11e74045f8a9d7033c5ead0c14 P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:26.898566 15236 tablet_bootstrap.cc:654] T 84f1ab11e74045f8a9d7033c5ead0c14 P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:26.898649 15242 tablet_bootstrap.cc:492] T dc544319be0b484f8e9e5a3fbc40f530 P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:26.898865 13296 tablet_service.cc:1467] Processing CreateTablet for tablet 84f1ab11e74045f8a9d7033c5ead0c14 (DEFAULT_TABLE table=rebalance_test_table_19 [id=e6609c16048348029c01374cb5a68ce8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:26.899111 13296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84f1ab11e74045f8a9d7033c5ead0c14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:26.899844 15242 tablet_bootstrap.cc:654] T dc544319be0b484f8e9e5a3fbc40f530 P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:26.900861 15238 tablet_bootstrap.cc:492] T dc544319be0b484f8e9e5a3fbc40f530 P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:26.900981 15242 tablet_bootstrap.cc:492] T dc544319be0b484f8e9e5a3fbc40f530 P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:26.901093 15242 ts_tablet_manager.cc:1367] T dc544319be0b484f8e9e5a3fbc40f530 P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:56:26.901109 15236 tablet_bootstrap.cc:492] T 84f1ab11e74045f8a9d7033c5ead0c14 P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:26.901186 15236 ts_tablet_manager.cc:1367] T 84f1ab11e74045f8a9d7033c5ead0c14 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:56:26.901583 15242 raft_consensus.cc:348] T dc544319be0b484f8e9e5a3fbc40f530 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:26.901628 15236 raft_consensus.cc:348] T 84f1ab11e74045f8a9d7033c5ead0c14 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:26.901690 15242 raft_consensus.cc:374] T dc544319be0b484f8e9e5a3fbc40f530 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:26.901697 15238 tablet_bootstrap.cc:654] T dc544319be0b484f8e9e5a3fbc40f530 P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:26.901715 15236 raft_consensus.cc:374] T 84f1ab11e74045f8a9d7033c5ead0c14 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:26.901719 15242 raft_consensus.cc:729] T dc544319be0b484f8e9e5a3fbc40f530 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:26.901743 15236 raft_consensus.cc:729] T 84f1ab11e74045f8a9d7033c5ead0c14 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:26.901867 15236 consensus_queue.cc:260] T 84f1ab11e74045f8a9d7033c5ead0c14 P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:26.902163 15236 ts_tablet_manager.cc:1398] T 84f1ab11e74045f8a9d7033c5ead0c14 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:26.902371 15242 consensus_queue.cc:260] T dc544319be0b484f8e9e5a3fbc40f530 P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:26.902776 15242 ts_tablet_manager.cc:1398] T dc544319be0b484f8e9e5a3fbc40f530 P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:26.902856 13165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84f1ab11e74045f8a9d7033c5ead0c14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:26.902954 15242 tablet_bootstrap.cc:492] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:26.903954 15242 tablet_bootstrap.cc:654] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:26.904170 15237 tablet_bootstrap.cc:492] T dc544319be0b484f8e9e5a3fbc40f530 P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:26.905138 15242 tablet_bootstrap.cc:492] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:26.905179 15237 tablet_bootstrap.cc:654] T dc544319be0b484f8e9e5a3fbc40f530 P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:26.905261 15242 ts_tablet_manager.cc:1367] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:56:26.905701 15242 raft_consensus.cc:348] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:26.905797 15242 raft_consensus.cc:374] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:26.905823 15242 raft_consensus.cc:729] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:26.905954 15242 consensus_queue.cc:260] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:26.906455 15242 ts_tablet_manager.cc:1398] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:26.898660 13295 tablet_service.cc:1467] Processing CreateTablet for tablet fd6a3d908840456f82c346c4bf1d8160 (DEFAULT_TABLE table=rebalance_test_table_19 [id=e6609c16048348029c01374cb5a68ce8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:26.907063 13295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd6a3d908840456f82c346c4bf1d8160. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:26.909301 15242 tablet_bootstrap.cc:492] T fd6a3d908840456f82c346c4bf1d8160 P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:26.910079 15238 tablet_bootstrap.cc:492] T dc544319be0b484f8e9e5a3fbc40f530 P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:26.910179 15238 ts_tablet_manager.cc:1367] T dc544319be0b484f8e9e5a3fbc40f530 P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.001s
I20240503 13:56:26.910645 15242 tablet_bootstrap.cc:654] T fd6a3d908840456f82c346c4bf1d8160 P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:26.910660 15238 raft_consensus.cc:348] T dc544319be0b484f8e9e5a3fbc40f530 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:26.910785 15238 raft_consensus.cc:374] T dc544319be0b484f8e9e5a3fbc40f530 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:26.910822 15238 raft_consensus.cc:729] T dc544319be0b484f8e9e5a3fbc40f530 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:26.910929 15238 consensus_queue.cc:260] T dc544319be0b484f8e9e5a3fbc40f530 P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:26.911196 15238 ts_tablet_manager.cc:1398] T dc544319be0b484f8e9e5a3fbc40f530 P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 13:56:26.911377 15238 tablet_bootstrap.cc:492] T fd6a3d908840456f82c346c4bf1d8160 P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:26.912128 15238 tablet_bootstrap.cc:654] T fd6a3d908840456f82c346c4bf1d8160 P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:26.912302 15242 tablet_bootstrap.cc:492] T fd6a3d908840456f82c346c4bf1d8160 P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:26.912374 15242 ts_tablet_manager.cc:1367] T fd6a3d908840456f82c346c4bf1d8160 P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:56:26.913089 15237 tablet_bootstrap.cc:492] T dc544319be0b484f8e9e5a3fbc40f530 P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:26.913182 15237 ts_tablet_manager.cc:1367] T dc544319be0b484f8e9e5a3fbc40f530 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.001s
I20240503 13:56:26.913643 15237 raft_consensus.cc:348] T dc544319be0b484f8e9e5a3fbc40f530 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:26.913751 15237 raft_consensus.cc:374] T dc544319be0b484f8e9e5a3fbc40f530 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:26.913782 15237 raft_consensus.cc:729] T dc544319be0b484f8e9e5a3fbc40f530 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:26.913887 15237 consensus_queue.cc:260] T dc544319be0b484f8e9e5a3fbc40f530 P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:26.914057 15220 raft_consensus.cc:482] T dc544319be0b484f8e9e5a3fbc40f530 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:26.914115 15220 raft_consensus.cc:504] T dc544319be0b484f8e9e5a3fbc40f530 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:26.914232 15237 ts_tablet_manager.cc:1398] T dc544319be0b484f8e9e5a3fbc40f530 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:26.914810 15220 leader_election.cc:288] T dc544319be0b484f8e9e5a3fbc40f530 P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695), ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:26.915630 15237 tablet_bootstrap.cc:492] T 84f1ab11e74045f8a9d7033c5ead0c14 P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:26.916599 15152 raft_consensus.cc:482] T dc544319be0b484f8e9e5a3fbc40f530 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:26.916688 15152 raft_consensus.cc:504] T dc544319be0b484f8e9e5a3fbc40f530 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:26.917042 15152 leader_election.cc:288] T dc544319be0b484f8e9e5a3fbc40f530 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387), 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:26.917127 15237 tablet_bootstrap.cc:654] T 84f1ab11e74045f8a9d7033c5ead0c14 P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:26.917565 13052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc544319be0b484f8e9e5a3fbc40f530" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e" is_pre_election: true
I20240503 13:56:26.917690 13052 raft_consensus.cc:2454] T dc544319be0b484f8e9e5a3fbc40f530 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 0.
I20240503 13:56:26.918102 13250 leader_election.cc:302] T dc544319be0b484f8e9e5a3fbc40f530 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:26.918431 13183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc544319be0b484f8e9e5a3fbc40f530" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770" is_pre_election: true
I20240503 13:56:26.918541 13183 raft_consensus.cc:2454] T dc544319be0b484f8e9e5a3fbc40f530 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 0.
I20240503 13:56:26.918583 15152 raft_consensus.cc:2781] T dc544319be0b484f8e9e5a3fbc40f530 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:26.918640 15152 raft_consensus.cc:482] T dc544319be0b484f8e9e5a3fbc40f530 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:26.918673 15152 raft_consensus.cc:3037] T dc544319be0b484f8e9e5a3fbc40f530 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:26.918861 15242 raft_consensus.cc:348] T fd6a3d908840456f82c346c4bf1d8160 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:26.918977 15242 raft_consensus.cc:374] T fd6a3d908840456f82c346c4bf1d8160 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:26.919011 15242 raft_consensus.cc:729] T fd6a3d908840456f82c346c4bf1d8160 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:26.919113 15242 consensus_queue.cc:260] T fd6a3d908840456f82c346c4bf1d8160 P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:26.919289 13316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc544319be0b484f8e9e5a3fbc40f530" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703" is_pre_election: true
I20240503 13:56:26.919759 15242 ts_tablet_manager.cc:1398] T fd6a3d908840456f82c346c4bf1d8160 P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240503 13:56:26.919742 15152 raft_consensus.cc:504] T dc544319be0b484f8e9e5a3fbc40f530 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:26.920248 15152 leader_election.cc:288] T dc544319be0b484f8e9e5a3fbc40f530 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: Requested vote from peers 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387), 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:26.920522 13316 raft_consensus.cc:2379] T dc544319be0b484f8e9e5a3fbc40f530 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8ccf5a4ac450414c9d3579bd5800b770 in current term 1: Already voted for candidate ba0194098aa84449ae17e8a0d4c43703 in this term.
I20240503 13:56:26.920667 13052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc544319be0b484f8e9e5a3fbc40f530" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e"
I20240503 13:56:26.920758 13183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc544319be0b484f8e9e5a3fbc40f530" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770"
I20240503 13:56:26.920794 13052 raft_consensus.cc:3037] T dc544319be0b484f8e9e5a3fbc40f530 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:26.920850 13183 raft_consensus.cc:3037] T dc544319be0b484f8e9e5a3fbc40f530 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:26.920861 15236 tablet_bootstrap.cc:492] T fd6a3d908840456f82c346c4bf1d8160 P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:26.921725 15236 tablet_bootstrap.cc:654] T fd6a3d908840456f82c346c4bf1d8160 P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:26.921730 13052 raft_consensus.cc:2454] T dc544319be0b484f8e9e5a3fbc40f530 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 1.
I20240503 13:56:26.921730 13183 raft_consensus.cc:2454] T dc544319be0b484f8e9e5a3fbc40f530 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 1.
I20240503 13:56:26.922055 13250 leader_election.cc:302] T dc544319be0b484f8e9e5a3fbc40f530 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:26.922282 15152 raft_consensus.cc:2781] T dc544319be0b484f8e9e5a3fbc40f530 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:26.922355 15152 raft_consensus.cc:686] T dc544319be0b484f8e9e5a3fbc40f530 P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Becoming Leader. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Running, Role: LEADER
I20240503 13:56:26.922472 15152 consensus_queue.cc:237] T dc544319be0b484f8e9e5a3fbc40f530 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:26.922636 15237 tablet_bootstrap.cc:492] T 84f1ab11e74045f8a9d7033c5ead0c14 P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:26.922722 15237 ts_tablet_manager.cc:1367] T 84f1ab11e74045f8a9d7033c5ead0c14 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240503 13:56:26.923224 15237 raft_consensus.cc:348] T 84f1ab11e74045f8a9d7033c5ead0c14 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:26.923316 15237 raft_consensus.cc:374] T 84f1ab11e74045f8a9d7033c5ead0c14 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:26.923345 15237 raft_consensus.cc:729] T 84f1ab11e74045f8a9d7033c5ead0c14 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:26.923475 15237 consensus_queue.cc:260] T 84f1ab11e74045f8a9d7033c5ead0c14 P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:26.923872 15237 ts_tablet_manager.cc:1398] T 84f1ab11e74045f8a9d7033c5ead0c14 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:26.923908 13053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc544319be0b484f8e9e5a3fbc40f530" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e" is_pre_election: true
I20240503 13:56:26.924065 13053 raft_consensus.cc:2379] T dc544319be0b484f8e9e5a3fbc40f530 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8ccf5a4ac450414c9d3579bd5800b770 in current term 1: Already voted for candidate ba0194098aa84449ae17e8a0d4c43703 in this term.
I20240503 13:56:26.924325 13119 leader_election.cc:302] T dc544319be0b484f8e9e5a3fbc40f530 P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ccf5a4ac450414c9d3579bd5800b770; no voters: 60ba3ffe3b7746469834cd91c84be14e, ba0194098aa84449ae17e8a0d4c43703
I20240503 13:56:26.924790 15238 tablet_bootstrap.cc:492] T fd6a3d908840456f82c346c4bf1d8160 P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:26.924885 15238 ts_tablet_manager.cc:1367] T fd6a3d908840456f82c346c4bf1d8160 P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.001s
I20240503 13:56:26.925437 15238 raft_consensus.cc:348] T fd6a3d908840456f82c346c4bf1d8160 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:26.925534 15238 raft_consensus.cc:374] T fd6a3d908840456f82c346c4bf1d8160 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:26.925567 15238 raft_consensus.cc:729] T fd6a3d908840456f82c346c4bf1d8160 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:26.925679 15238 consensus_queue.cc:260] T fd6a3d908840456f82c346c4bf1d8160 P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:26.926004 15238 ts_tablet_manager.cc:1398] T fd6a3d908840456f82c346c4bf1d8160 P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:26.927028 15345 raft_consensus.cc:2726] T dc544319be0b484f8e9e5a3fbc40f530 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 13:56:26.929731 12907 catalog_manager.cc:5484] T dc544319be0b484f8e9e5a3fbc40f530 P ba0194098aa84449ae17e8a0d4c43703 reported cstate change: term changed from 0 to 1, leader changed from <none> to ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67). New cstate: current_term: 1 leader_uuid: "ba0194098aa84449ae17e8a0d4c43703" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:26.933266 15236 tablet_bootstrap.cc:492] T fd6a3d908840456f82c346c4bf1d8160 P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:26.933368 15236 ts_tablet_manager.cc:1367] T fd6a3d908840456f82c346c4bf1d8160 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 13:56:26.933804 15236 raft_consensus.cc:348] T fd6a3d908840456f82c346c4bf1d8160 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:26.933928 15236 raft_consensus.cc:374] T fd6a3d908840456f82c346c4bf1d8160 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:26.933959 15236 raft_consensus.cc:729] T fd6a3d908840456f82c346c4bf1d8160 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:26.934068 15236 consensus_queue.cc:260] T fd6a3d908840456f82c346c4bf1d8160 P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:26.934608 15236 ts_tablet_manager.cc:1398] T fd6a3d908840456f82c346c4bf1d8160 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:27.032006 15206 raft_consensus.cc:482] T fd6a3d908840456f82c346c4bf1d8160 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:27.032136 15206 raft_consensus.cc:504] T fd6a3d908840456f82c346c4bf1d8160 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:27.032969 15206 leader_election.cc:288] T fd6a3d908840456f82c346c4bf1d8160 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695), 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:27.033282 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd6a3d908840456f82c346c4bf1d8160" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
I20240503 13:56:27.033277 13053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd6a3d908840456f82c346c4bf1d8160" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e" is_pre_election: true
I20240503 13:56:27.033407 13053 raft_consensus.cc:2454] T fd6a3d908840456f82c346c4bf1d8160 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 0.
I20240503 13:56:27.033412 13446 raft_consensus.cc:2454] T fd6a3d908840456f82c346c4bf1d8160 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 0.
I20240503 13:56:27.033697 13250 leader_election.cc:302] T fd6a3d908840456f82c346c4bf1d8160 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:27.033883 15206 raft_consensus.cc:2781] T fd6a3d908840456f82c346c4bf1d8160 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:27.033953 15206 raft_consensus.cc:482] T fd6a3d908840456f82c346c4bf1d8160 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:27.033982 15206 raft_consensus.cc:3037] T fd6a3d908840456f82c346c4bf1d8160 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:27.034848 15206 raft_consensus.cc:504] T fd6a3d908840456f82c346c4bf1d8160 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:27.035154 15206 leader_election.cc:288] T fd6a3d908840456f82c346c4bf1d8160 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: Requested vote from peers 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695), 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:27.035485 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd6a3d908840456f82c346c4bf1d8160" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
I20240503 13:56:27.035481 13053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd6a3d908840456f82c346c4bf1d8160" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e"
I20240503 13:56:27.035609 13446 raft_consensus.cc:3037] T fd6a3d908840456f82c346c4bf1d8160 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:27.035609 13053 raft_consensus.cc:3037] T fd6a3d908840456f82c346c4bf1d8160 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:27.036603 13446 raft_consensus.cc:2454] T fd6a3d908840456f82c346c4bf1d8160 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 1.
I20240503 13:56:27.036604 13053 raft_consensus.cc:2454] T fd6a3d908840456f82c346c4bf1d8160 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 1.
I20240503 13:56:27.036899 13250 leader_election.cc:302] T fd6a3d908840456f82c346c4bf1d8160 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:27.037074 15206 raft_consensus.cc:2781] T fd6a3d908840456f82c346c4bf1d8160 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:27.037153 15206 raft_consensus.cc:686] T fd6a3d908840456f82c346c4bf1d8160 P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Becoming Leader. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Running, Role: LEADER
I20240503 13:56:27.037285 15206 consensus_queue.cc:237] T fd6a3d908840456f82c346c4bf1d8160 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:27.039307 12907 catalog_manager.cc:5484] T fd6a3d908840456f82c346c4bf1d8160 P ba0194098aa84449ae17e8a0d4c43703 reported cstate change: term changed from 0 to 1, leader changed from <none> to ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67). New cstate: current_term: 1 leader_uuid: "ba0194098aa84449ae17e8a0d4c43703" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } }
I20240503 13:56:27.232537 15206 raft_consensus.cc:482] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:27.232667 15206 raft_consensus.cc:504] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:27.233050 15206 leader_election.cc:288] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:27.233374 13183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f1ab11e74045f8a9d7033c5ead0c14" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770" is_pre_election: true
I20240503 13:56:27.233501 13183 raft_consensus.cc:2454] T 84f1ab11e74045f8a9d7033c5ead0c14 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 0.
I20240503 13:56:27.233456 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f1ab11e74045f8a9d7033c5ead0c14" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
I20240503 13:56:27.233606 13446 raft_consensus.cc:2454] T 84f1ab11e74045f8a9d7033c5ead0c14 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 0.
I20240503 13:56:27.233934 13249 leader_election.cc:302] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:27.234115 15206 raft_consensus.cc:2781] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:27.234170 15206 raft_consensus.cc:482] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:27.234215 15206 raft_consensus.cc:3037] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:27.235060 15206 raft_consensus.cc:504] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:27.235384 15206 leader_election.cc:288] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: Requested vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:27.235702 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f1ab11e74045f8a9d7033c5ead0c14" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
I20240503 13:56:27.235698 13183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f1ab11e74045f8a9d7033c5ead0c14" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770"
I20240503 13:56:27.235814 13183 raft_consensus.cc:3037] T 84f1ab11e74045f8a9d7033c5ead0c14 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:27.235814 13446 raft_consensus.cc:3037] T 84f1ab11e74045f8a9d7033c5ead0c14 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:27.236783 13183 raft_consensus.cc:2454] T 84f1ab11e74045f8a9d7033c5ead0c14 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 1.
I20240503 13:56:27.236783 13446 raft_consensus.cc:2454] T 84f1ab11e74045f8a9d7033c5ead0c14 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 1.
I20240503 13:56:27.237061 13249 leader_election.cc:302] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:27.237197 15206 raft_consensus.cc:2781] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:27.237282 15206 raft_consensus.cc:686] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Becoming Leader. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Running, Role: LEADER
I20240503 13:56:27.237418 15206 consensus_queue.cc:237] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:27.239459 12907 catalog_manager.cc:5484] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 reported cstate change: term changed from 0 to 1, leader changed from <none> to ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67). New cstate: current_term: 1 leader_uuid: "ba0194098aa84449ae17e8a0d4c43703" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } }
W20240503 13:56:27.404382 15352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:56:27.404711 15352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:56:27.407982 15352 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_19'
I20240503 13:56:27.493983 15152 consensus_queue.cc:1035] T dc544319be0b484f8e9e5a3fbc40f530 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:27.499603 15154 consensus_queue.cc:1035] T dc544319be0b484f8e9e5a3fbc40f530 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:27.500854 13446 raft_consensus.cc:1261] T fd6a3d908840456f82c346c4bf1d8160 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:27.502449 15154 consensus_queue.cc:1035] T fd6a3d908840456f82c346c4bf1d8160 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:27.503188 13184 raft_consensus.cc:1261] T 84f1ab11e74045f8a9d7033c5ead0c14 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:27.503192 13053 raft_consensus.cc:1261] T fd6a3d908840456f82c346c4bf1d8160 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:27.503554 15152 consensus_queue.cc:1035] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:27.503607 15154 consensus_queue.cc:1035] T fd6a3d908840456f82c346c4bf1d8160 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:27.504446 13447 raft_consensus.cc:1261] T 84f1ab11e74045f8a9d7033c5ead0c14 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:27.507272 15152 consensus_queue.cc:1035] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:27.521512 15383 mvcc.cc:205] Tried to move back new op lower bound from 7023593830399614976 to 7023593828506157056. Current Snapshot: MvccSnapshot[applied={T|T < 7023593830399614976}]
INSERT report
rows total: 2000
time total: 123.186 ms
time per row: 0.0615931 ms
I20240503 13:56:27.577605 6537 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
/tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.98.69:39153
--local_ip_for_outbound_sockets=127.6.98.69
--tserver_master_addrs=127.6.98.126:44613
--webserver_port=44007
--webserver_interface=127.6.98.69
--builtin_ntp_servers=127.6.98.84:39449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 13:56:27.693868 15411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:56:27.694164 15411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:56:27.694221 15411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:56:27.697521 15411 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 13:56:27.697594 15411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:56:27.697682 15411 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.98.69
I20240503 13:56:27.701511 15411 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.98.84:39449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.98.69:39153
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.6.98.69
--webserver_port=44007
--tserver_master_addrs=127.6.98.126:44613
--never_fsync=true
--heap_profile_path=/tmp/kudu.15411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.98.69
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:56:27.702558 15411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:56:27.703776 15411 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:56:27.713464 15417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:56:27.713763 15411 server_base.cc:1034] running on GCE node
W20240503 13:56:27.713917 15419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:56:27.714237 15416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:56:27.714778 15411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:56:27.715533 15411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:56:27.716724 15411 hybrid_clock.cc:648] HybridClock initialized: now 1714744587716711 us; error 30 us; skew 500 ppm
I20240503 13:56:27.719029 15411 webserver.cc:460] Webserver started at http://127.6.98.69:44007/ using document root <none> and password file <none>
I20240503 13:56:27.719713 15411 fs_manager.cc:359] Metadata directory not provided
I20240503 13:56:27.719799 15411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:56:27.724479 15411 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 13:56:27.727123 15425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:56:27.728533 15411 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 13:56:27.728669 15411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "61b887f0d40d497badeca5604c1f0546"
format_stamp: "Formatted at 2024-05-03 13:56:11 on dist-test-slave-zg9z"
I20240503 13:56:27.729195 15411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:56:27.774206 15411 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:56:27.774902 15411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:56:27.775069 15411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:56:27.775630 15411 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:56:27.776746 15411 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:56:27.776806 15411 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:56:27.776862 15411 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:56:27.776891 15411 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:56:27.786288 15538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.98.69:39153 every 8 connection(s)
I20240503 13:56:27.786260 15411 rpc_server.cc:307] RPC server started. Bound to: 127.6.98.69:39153
I20240503 13:56:27.787509 15411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 13:56:27.794435 6537 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu as pid 15411
I20240503 13:56:27.794894 6537 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
/tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.98.70:34951
--local_ip_for_outbound_sockets=127.6.98.70
--tserver_master_addrs=127.6.98.126:44613
--webserver_port=44359
--webserver_interface=127.6.98.70
--builtin_ntp_servers=127.6.98.84:39449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 13:56:27.797423 15539 heartbeater.cc:347] Connected to a master server at 127.6.98.126:44613
I20240503 13:56:27.797675 15539 heartbeater.cc:464] Registering TS with master...
I20240503 13:56:27.798233 15539 heartbeater.cc:511] Master 127.6.98.126:44613 requested a full tablet report, sending...
I20240503 13:56:27.799360 12909 ts_manager.cc:194] Re-registered known tserver with Master: 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153)
I20240503 13:56:27.799947 12909 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.98.69:38569
W20240503 13:56:27.909055 15542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:56:27.909844 15542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:56:27.909960 15542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:56:27.914312 15542 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 13:56:27.914409 15542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:56:27.914510 15542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.98.70
I20240503 13:56:27.919801 15542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.98.84:39449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.98.70:34951
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.6.98.70
--webserver_port=44359
--tserver_master_addrs=127.6.98.126:44613
--never_fsync=true
--heap_profile_path=/tmp/kudu.15542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.98.70
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:56:27.921053 15542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:56:27.922410 15542 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:56:27.932332 15548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:56:27.932607 15542 server_base.cc:1034] running on GCE node
W20240503 13:56:27.932849 15551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:56:27.939031 15549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:56:27.939550 15542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:56:27.940327 15542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:56:27.941448 15542 hybrid_clock.cc:648] HybridClock initialized: now 1714744587941433 us; error 31 us; skew 500 ppm
I20240503 13:56:27.943418 15542 webserver.cc:460] Webserver started at http://127.6.98.70:44359/ using document root <none> and password file <none>
I20240503 13:56:27.944036 15542 fs_manager.cc:359] Metadata directory not provided
I20240503 13:56:27.944110 15542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:56:27.947839 15542 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:56:27.949784 15557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:56:27.950745 15542 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:27.950839 15542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "009f849ee7b54c80a3f5264695047e4d"
format_stamp: "Formatted at 2024-05-03 13:56:11 on dist-test-slave-zg9z"
I20240503 13:56:27.951272 15542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:56:27.966924 15542 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:56:27.967605 15542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:56:27.967756 15542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:56:27.968304 15542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:56:27.969338 15542 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:56:27.969386 15542 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:56:27.969439 15542 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:56:27.969466 15542 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:56:27.978277 15542 rpc_server.cc:307] RPC server started. Bound to: 127.6.98.70:34951
I20240503 13:56:27.978297 15670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.98.70:34951 every 8 connection(s)
I20240503 13:56:27.979292 15542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 13:56:27.981689 6537 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu as pid 15542
I20240503 13:56:27.982286 6537 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
/tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.98.71:42885
--local_ip_for_outbound_sockets=127.6.98.71
--tserver_master_addrs=127.6.98.126:44613
--webserver_port=45699
--webserver_interface=127.6.98.71
--builtin_ntp_servers=127.6.98.84:39449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 13:56:27.989382 15671 heartbeater.cc:347] Connected to a master server at 127.6.98.126:44613
I20240503 13:56:27.989676 15671 heartbeater.cc:464] Registering TS with master...
I20240503 13:56:27.990290 15671 heartbeater.cc:511] Master 127.6.98.126:44613 requested a full tablet report, sending...
I20240503 13:56:27.991416 12909 ts_manager.cc:194] Re-registered known tserver with Master: 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951)
I20240503 13:56:27.992038 12909 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.98.70:33371
W20240503 13:56:28.104234 15674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:56:28.104616 15674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:56:28.104707 15674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:56:28.109059 15674 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 13:56:28.109143 15674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:56:28.109243 15674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.98.71
I20240503 13:56:28.114321 15674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.98.84:39449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.98.71:42885
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.6.98.71
--webserver_port=45699
--tserver_master_addrs=127.6.98.126:44613
--never_fsync=true
--heap_profile_path=/tmp/kudu.15674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.98.71
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:56:28.115419 15674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:56:28.116782 15674 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:56:28.126080 15681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:56:28.126099 15680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:56:28.126318 15683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:56:28.126673 15674 server_base.cc:1034] running on GCE node
I20240503 13:56:28.127533 15674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:56:28.128150 15674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:56:28.129340 15674 hybrid_clock.cc:648] HybridClock initialized: now 1714744588129310 us; error 42 us; skew 500 ppm
I20240503 13:56:28.131105 15674 webserver.cc:460] Webserver started at http://127.6.98.71:45699/ using document root <none> and password file <none>
I20240503 13:56:28.131628 15674 fs_manager.cc:359] Metadata directory not provided
I20240503 13:56:28.131692 15674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:56:28.135393 15674 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20240503 13:56:28.137377 15689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:56:28.138679 15674 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:56:28.138785 15674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "76c386bef61a4bd5a907266a5118ffe2"
format_stamp: "Formatted at 2024-05-03 13:56:11 on dist-test-slave-zg9z"
I20240503 13:56:28.139261 15674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:56:28.158672 15674 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:56:28.159317 15674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:56:28.159478 15674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:56:28.160089 15674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:56:28.161163 15674 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:56:28.161223 15674 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:56:28.161291 15674 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:56:28.161320 15674 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:56:28.170333 15674 rpc_server.cc:307] RPC server started. Bound to: 127.6.98.71:42885
I20240503 13:56:28.170370 15803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.98.71:42885 every 8 connection(s)
I20240503 13:56:28.171340 15674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 13:56:28.179586 15804 heartbeater.cc:347] Connected to a master server at 127.6.98.126:44613
I20240503 13:56:28.179811 15804 heartbeater.cc:464] Registering TS with master...
I20240503 13:56:28.180076 6537 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu as pid 15674
I20240503 13:56:28.180289 15804 heartbeater.cc:511] Master 127.6.98.126:44613 requested a full tablet report, sending...
I20240503 13:56:28.180449 6537 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
/tmp/dist-test-taskFG4o4C/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.98.72:44753
--local_ip_for_outbound_sockets=127.6.98.72
--tserver_master_addrs=127.6.98.126:44613
--webserver_port=45043
--webserver_interface=127.6.98.72
--builtin_ntp_servers=127.6.98.84:39449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 13:56:28.181282 12909 ts_manager.cc:194] Re-registered known tserver with Master: 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885)
I20240503 13:56:28.181964 12909 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.98.71:49827
W20240503 13:56:28.292254 15807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:56:28.292465 15807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:56:28.292537 15807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:56:28.295593 15807 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 13:56:28.295657 15807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:56:28.295724 15807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.98.72
I20240503 13:56:28.300997 15807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.98.84:39449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.98.72:44753
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.6.98.72
--webserver_port=45043
--tserver_master_addrs=127.6.98.126:44613
--never_fsync=true
--heap_profile_path=/tmp/kudu.15807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.98.72
--log_dir=/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:56:28.302356 15807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:56:28.304095 15807 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:56:28.314083 15813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:56:28.314486 15815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:56:28.314615 15812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:56:28.315021 15807 server_base.cc:1034] running on GCE node
I20240503 13:56:28.315408 15807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:56:28.315994 15807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:56:28.317157 15807 hybrid_clock.cc:648] HybridClock initialized: now 1714744588317141 us; error 33 us; skew 500 ppm
I20240503 13:56:28.319307 15807 webserver.cc:460] Webserver started at http://127.6.98.72:45043/ using document root <none> and password file <none>
I20240503 13:56:28.319883 15807 fs_manager.cc:359] Metadata directory not provided
I20240503 13:56:28.319962 15807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:56:28.323945 15807 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20240503 13:56:28.326483 15821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:56:28.327667 15807 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240503 13:56:28.327814 15807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "e47105a0f46e4511b7112e571688a4c7"
format_stamp: "Formatted at 2024-05-03 13:56:11 on dist-test-slave-zg9z"
I20240503 13:56:28.328264 15807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:56:28.351902 15807 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:56:28.352577 15807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:56:28.352739 15807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:56:28.353327 15807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:56:28.354250 15807 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:56:28.354347 15807 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:56:28.354533 15807 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:56:28.354584 15807 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:56:28.363750 15807 rpc_server.cc:307] RPC server started. Bound to: 127.6.98.72:44753
I20240503 13:56:28.363806 15934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.98.72:44753 every 8 connection(s)
I20240503 13:56:28.364836 15807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 13:56:28.367841 6537 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFG4o4C/build/debug/bin/kudu as pid 15807
I20240503 13:56:28.371119 12909 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:56:28.374128 15935 heartbeater.cc:347] Connected to a master server at 127.6.98.126:44613
I20240503 13:56:28.374357 15935 heartbeater.cc:464] Registering TS with master...
I20240503 13:56:28.374884 15935 heartbeater.cc:511] Master 127.6.98.126:44613 requested a full tablet report, sending...
I20240503 13:56:28.376425 12909 ts_manager.cc:194] Re-registered known tserver with Master: e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:28.377146 12909 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.98.72:45279
I20240503 13:56:28.377755 13033 tablet_service.cc:1467] Processing CreateTablet for tablet 4ab2e8e604954be39ae6b8f1e028c4ba (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d6641e092440471ba33512c2cb1d2d2b]), partition=
I20240503 13:56:28.378050 13033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ab2e8e604954be39ae6b8f1e028c4ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:28.380379 15950 tablet_bootstrap.cc:492] T 4ab2e8e604954be39ae6b8f1e028c4ba P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:28.380916 13295 tablet_service.cc:1467] Processing CreateTablet for tablet 4ab2e8e604954be39ae6b8f1e028c4ba (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d6641e092440471ba33512c2cb1d2d2b]), partition=
I20240503 13:56:28.381183 13295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ab2e8e604954be39ae6b8f1e028c4ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:28.381280 15950 tablet_bootstrap.cc:654] T 4ab2e8e604954be39ae6b8f1e028c4ba P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:28.382146 15950 tablet_bootstrap.cc:492] T 4ab2e8e604954be39ae6b8f1e028c4ba P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:28.382233 15950 ts_tablet_manager.cc:1367] T 4ab2e8e604954be39ae6b8f1e028c4ba P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:28.382607 15950 raft_consensus.cc:348] T 4ab2e8e604954be39ae6b8f1e028c4ba P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:28.382702 15950 raft_consensus.cc:374] T 4ab2e8e604954be39ae6b8f1e028c4ba P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:28.382731 15950 raft_consensus.cc:729] T 4ab2e8e604954be39ae6b8f1e028c4ba P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:28.382875 15950 consensus_queue.cc:260] T 4ab2e8e604954be39ae6b8f1e028c4ba P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:28.383126 15950 ts_tablet_manager.cc:1398] T 4ab2e8e604954be39ae6b8f1e028c4ba P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:28.383391 15952 tablet_bootstrap.cc:492] T 4ab2e8e604954be39ae6b8f1e028c4ba P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:28.384243 15952 tablet_bootstrap.cc:654] T 4ab2e8e604954be39ae6b8f1e028c4ba P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:28.385833 15952 tablet_bootstrap.cc:492] T 4ab2e8e604954be39ae6b8f1e028c4ba P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:28.386199 15952 ts_tablet_manager.cc:1367] T 4ab2e8e604954be39ae6b8f1e028c4ba P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:56:28.386603 15952 raft_consensus.cc:348] T 4ab2e8e604954be39ae6b8f1e028c4ba P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:28.386730 15952 raft_consensus.cc:374] T 4ab2e8e604954be39ae6b8f1e028c4ba P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:28.386772 15952 raft_consensus.cc:729] T 4ab2e8e604954be39ae6b8f1e028c4ba P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:28.386904 15952 consensus_queue.cc:260] T 4ab2e8e604954be39ae6b8f1e028c4ba P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:28.387352 15952 ts_tablet_manager.cc:1398] T 4ab2e8e604954be39ae6b8f1e028c4ba P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:28.399637 15738 tablet_service.cc:1467] Processing CreateTablet for tablet 4ab2e8e604954be39ae6b8f1e028c4ba (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d6641e092440471ba33512c2cb1d2d2b]), partition=
I20240503 13:56:28.400444 15738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ab2e8e604954be39ae6b8f1e028c4ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:28.407589 15954 tablet_bootstrap.cc:492] T 4ab2e8e604954be39ae6b8f1e028c4ba P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:28.409826 15954 tablet_bootstrap.cc:654] T 4ab2e8e604954be39ae6b8f1e028c4ba P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:28.410581 15954 log.cc:824] T 4ab2e8e604954be39ae6b8f1e028c4ba P 76c386bef61a4bd5a907266a5118ffe2: Log is configured to *not* fsync() on all Append() calls
I20240503 13:56:28.412263 15954 tablet_bootstrap.cc:492] T 4ab2e8e604954be39ae6b8f1e028c4ba P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:28.412528 15954 ts_tablet_manager.cc:1367] T 4ab2e8e604954be39ae6b8f1e028c4ba P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20240503 13:56:28.415921 15954 raft_consensus.cc:348] T 4ab2e8e604954be39ae6b8f1e028c4ba P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:28.416142 15954 raft_consensus.cc:374] T 4ab2e8e604954be39ae6b8f1e028c4ba P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:28.416191 15954 raft_consensus.cc:729] T 4ab2e8e604954be39ae6b8f1e028c4ba P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:28.416661 15954 consensus_queue.cc:260] T 4ab2e8e604954be39ae6b8f1e028c4ba P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:28.417539 15954 ts_tablet_manager.cc:1398] T 4ab2e8e604954be39ae6b8f1e028c4ba P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240503 13:56:28.417718 15804 heartbeater.cc:503] Master 127.6.98.126:44613 was elected leader, sending a full tablet report...
W20240503 13:56:28.422123 15805 tablet.cc:2369] T 4ab2e8e604954be39ae6b8f1e028c4ba P 76c386bef61a4bd5a907266a5118ffe2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 13:56:28.477629 15332 raft_consensus.cc:482] T 4ab2e8e604954be39ae6b8f1e028c4ba P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:28.477747 15332 raft_consensus.cc:504] T 4ab2e8e604954be39ae6b8f1e028c4ba P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:28.478216 15332 leader_election.cc:288] T 4ab2e8e604954be39ae6b8f1e028c4ba P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885), ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:28.478987 13316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ab2e8e604954be39ae6b8f1e028c4ba" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703" is_pre_election: true
I20240503 13:56:28.479184 13316 raft_consensus.cc:2454] T 4ab2e8e604954be39ae6b8f1e028c4ba P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 0.
I20240503 13:56:28.479511 12990 leader_election.cc:302] T 4ab2e8e604954be39ae6b8f1e028c4ba P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:28.479684 15332 raft_consensus.cc:2781] T 4ab2e8e604954be39ae6b8f1e028c4ba P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:28.479739 15332 raft_consensus.cc:482] T 4ab2e8e604954be39ae6b8f1e028c4ba P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:28.479774 15332 raft_consensus.cc:3037] T 4ab2e8e604954be39ae6b8f1e028c4ba P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:28.480708 15332 raft_consensus.cc:504] T 4ab2e8e604954be39ae6b8f1e028c4ba P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:28.481060 15332 leader_election.cc:288] T 4ab2e8e604954be39ae6b8f1e028c4ba P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 election: Requested vote from peers 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885), ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:28.481474 13316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ab2e8e604954be39ae6b8f1e028c4ba" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703"
I20240503 13:56:28.481631 13316 raft_consensus.cc:3037] T 4ab2e8e604954be39ae6b8f1e028c4ba P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:28.482538 13316 raft_consensus.cc:2454] T 4ab2e8e604954be39ae6b8f1e028c4ba P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 1.
I20240503 13:56:28.482820 12990 leader_election.cc:302] T 4ab2e8e604954be39ae6b8f1e028c4ba P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:28.483067 15332 raft_consensus.cc:2781] T 4ab2e8e604954be39ae6b8f1e028c4ba P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:28.483260 15332 raft_consensus.cc:686] T 4ab2e8e604954be39ae6b8f1e028c4ba P 60ba3ffe3b7746469834cd91c84be14e [term 1 LEADER]: Becoming Leader. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Running, Role: LEADER
I20240503 13:56:28.483412 15332 consensus_queue.cc:237] T 4ab2e8e604954be39ae6b8f1e028c4ba P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:28.485903 15758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ab2e8e604954be39ae6b8f1e028c4ba" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2" is_pre_election: true
I20240503 13:56:28.486210 15758 raft_consensus.cc:2454] T 4ab2e8e604954be39ae6b8f1e028c4ba P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 0.
I20240503 13:56:28.486132 12905 catalog_manager.cc:5484] T 4ab2e8e604954be39ae6b8f1e028c4ba P 60ba3ffe3b7746469834cd91c84be14e reported cstate change: term changed from 0 to 1, leader changed from <none> to 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65). New cstate: current_term: 1 leader_uuid: "60ba3ffe3b7746469834cd91c84be14e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } }
I20240503 13:56:28.486553 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ab2e8e604954be39ae6b8f1e028c4ba" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2"
I20240503 13:56:28.486660 15757 raft_consensus.cc:3037] T 4ab2e8e604954be39ae6b8f1e028c4ba P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:28.487759 15757 raft_consensus.cc:2454] T 4ab2e8e604954be39ae6b8f1e028c4ba P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 1.
I20240503 13:56:28.609696 12906 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:56:28.615432 13314 raft_consensus.cc:1261] T 4ab2e8e604954be39ae6b8f1e028c4ba P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Refusing update from remote peer 60ba3ffe3b7746469834cd91c84be14e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:28.616439 15958 consensus_queue.cc:1035] T 4ab2e8e604954be39ae6b8f1e028c4ba P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:28.616585 15757 raft_consensus.cc:1261] T 4ab2e8e604954be39ae6b8f1e028c4ba P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Refusing update from remote peer 60ba3ffe3b7746469834cd91c84be14e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:28.618207 15332 consensus_queue.cc:1035] T 4ab2e8e604954be39ae6b8f1e028c4ba P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:28.620241 16004 tablet.cc:1722] T 4ab2e8e604954be39ae6b8f1e028c4ba P 60ba3ffe3b7746469834cd91c84be14e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:28.621191 12905 catalog_manager.cc:5576] TS ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279) does not have the latest schema for tablet 4ab2e8e604954be39ae6b8f1e028c4ba (table rebalancer_dynamic_table [id=d6641e092440471ba33512c2cb1d2d2b]). Expected version 1 got 0
I20240503 13:56:28.621330 16004 tablet.cc:1620] T 4ab2e8e604954be39ae6b8f1e028c4ba P 60ba3ffe3b7746469834cd91c84be14e: MemRowSet was empty: no flush needed.
I20240503 13:56:28.624477 12906 catalog_manager.cc:5576] TS ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279) does not have the latest schema for tablet 4ab2e8e604954be39ae6b8f1e028c4ba (table rebalancer_dynamic_table [id=d6641e092440471ba33512c2cb1d2d2b]). Expected version 1 got 0
I20240503 13:56:28.625526 12906 catalog_manager.cc:5576] TS 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885) does not have the latest schema for tablet 4ab2e8e604954be39ae6b8f1e028c4ba (table rebalancer_dynamic_table [id=d6641e092440471ba33512c2cb1d2d2b]). Expected version 1 got 0
I20240503 13:56:28.627813 16006 tablet.cc:1722] T 4ab2e8e604954be39ae6b8f1e028c4ba P ba0194098aa84449ae17e8a0d4c43703: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:28.628239 12906 catalog_manager.cc:5576] TS 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885) does not have the latest schema for tablet 4ab2e8e604954be39ae6b8f1e028c4ba (table rebalancer_dynamic_table [id=d6641e092440471ba33512c2cb1d2d2b]). Expected version 1 got 0
I20240503 13:56:28.634800 12906 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=d6641e092440471ba33512c2cb1d2d2b] alter complete (version 1)
I20240503 13:56:28.635915 16005 mvcc.cc:205] Tried to move back new op lower bound from 7023593834964959232 to 7023593834431082496. Current Snapshot: MvccSnapshot[applied={T|T < 7023593834964959232}]
I20240503 13:56:28.636101 16006 tablet.cc:1620] T 4ab2e8e604954be39ae6b8f1e028c4ba P ba0194098aa84449ae17e8a0d4c43703: MemRowSet was empty: no flush needed.
I20240503 13:56:28.636992 15473 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:56:28.643224 15738 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:56:28.646129 13426 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 13:56:28.656283 13165 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:56:28.659147 16026 tablet.cc:1722] T 4ab2e8e604954be39ae6b8f1e028c4ba P 76c386bef61a4bd5a907266a5118ffe2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:28.661470 15605 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:56:28.662613 15869 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:56:28.670142 16026 tablet.cc:1620] T 4ab2e8e604954be39ae6b8f1e028c4ba P 76c386bef61a4bd5a907266a5118ffe2: MemRowSet was empty: no flush needed.
I20240503 13:56:28.674234 13295 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240503 13:56:28.685012 13033 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 13:56:28.740139 12906 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:56:28.746750 16004 tablet.cc:1722] T 4ab2e8e604954be39ae6b8f1e028c4ba P 60ba3ffe3b7746469834cd91c84be14e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:28.747131 16004 tablet.cc:1620] T 4ab2e8e604954be39ae6b8f1e028c4ba P 60ba3ffe3b7746469834cd91c84be14e: MemRowSet was empty: no flush needed.
I20240503 13:56:28.747304 16026 tablet.cc:1722] T 4ab2e8e604954be39ae6b8f1e028c4ba P 76c386bef61a4bd5a907266a5118ffe2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:28.747596 16026 tablet.cc:1620] T 4ab2e8e604954be39ae6b8f1e028c4ba P 76c386bef61a4bd5a907266a5118ffe2: MemRowSet was empty: no flush needed.
I20240503 13:56:28.751442 16006 tablet.cc:1722] T 4ab2e8e604954be39ae6b8f1e028c4ba P ba0194098aa84449ae17e8a0d4c43703: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:28.751752 16006 tablet.cc:1620] T 4ab2e8e604954be39ae6b8f1e028c4ba P ba0194098aa84449ae17e8a0d4c43703: MemRowSet was empty: no flush needed.
I20240503 13:56:28.753079 12905 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=d6641e092440471ba33512c2cb1d2d2b] alter complete (version 2)
I20240503 13:56:28.801925 15539 heartbeater.cc:503] Master 127.6.98.126:44613 was elected leader, sending a full tablet report...
I20240503 13:56:28.839617 13033 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 13:56:28.841104 15605 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:56:28.848755 15473 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:56:28.849529 13426 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 13:56:28.850713 15738 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:56:28.859503 12906 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:56:28.862567 13295 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240503 13:56:28.862969 13165 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:56:28.863746 15869 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:56:28.867202 16004 tablet.cc:1722] T 4ab2e8e604954be39ae6b8f1e028c4ba P 60ba3ffe3b7746469834cd91c84be14e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:28.868136 16026 tablet.cc:1722] T 4ab2e8e604954be39ae6b8f1e028c4ba P 76c386bef61a4bd5a907266a5118ffe2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:28.869057 12906 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalance_test_table_0" } modify_external_catalogs: true
I20240503 13:56:28.869194 12906 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalance_test_table_0" } modify_external_catalogs: true
I20240503 13:56:28.870658 12912 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_extra_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:56:28.871986 16006 tablet.cc:1722] T 4ab2e8e604954be39ae6b8f1e028c4ba P ba0194098aa84449ae17e8a0d4c43703: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:28.876108 12909 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table_new_name [id=d6641e092440471ba33512c2cb1d2d2b] alter complete (version 3)
I20240503 13:56:28.878360 12906 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Sending DeleteTablet for 3 replicas of tablet 101417485ac24fd9bb868fd81b2552e5
I20240503 13:56:28.878917 12906 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Sending DeleteTablet for 3 replicas of tablet 5f79adb3e75c4a45998c73f29b65e312
I20240503 13:56:28.879400 12906 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Sending DeleteTablet for 3 replicas of tablet 3dd1e318c2084e3a9a80987731920ac8
I20240503 13:56:28.882201 15738 tablet_service.cc:1467] Processing CreateTablet for tablet 0878ccb6135f4485b5ba35ae67a76d0c (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=18c0e326c0a144c382cd95a8c0052057]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:28.882525 15737 tablet_service.cc:1467] Processing CreateTablet for tablet 356ccd6fdf25470dba2a79367a37128e (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=18c0e326c0a144c382cd95a8c0052057]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:28.882632 15738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0878ccb6135f4485b5ba35ae67a76d0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:28.882777 15737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 356ccd6fdf25470dba2a79367a37128e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:28.883734 13034 tablet_service.cc:1514] Processing DeleteTablet for tablet 3dd1e318c2084e3a9a80987731920ac8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:28 UTC) from {username='slave'} at 127.0.0.1:59458
I20240503 13:56:28.884570 13032 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f79adb3e75c4a45998c73f29b65e312 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:28 UTC) from {username='slave'} at 127.0.0.1:59458
I20240503 13:56:28.884743 13296 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f79adb3e75c4a45998c73f29b65e312 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:28 UTC) from {username='slave'} at 127.0.0.1:33250
I20240503 13:56:28.884975 15954 tablet_bootstrap.cc:492] T 356ccd6fdf25470dba2a79367a37128e P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:28.885500 13295 tablet_service.cc:1514] Processing DeleteTablet for tablet 101417485ac24fd9bb868fd81b2552e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:28 UTC) from {username='slave'} at 127.0.0.1:33250
I20240503 13:56:28.885645 13033 tablet_service.cc:1514] Processing DeleteTablet for tablet 101417485ac24fd9bb868fd81b2552e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:28 UTC) from {username='slave'} at 127.0.0.1:59458
I20240503 13:56:28.885845 15954 tablet_bootstrap.cc:654] T 356ccd6fdf25470dba2a79367a37128e P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:28.885960 13425 tablet_service.cc:1514] Processing DeleteTablet for tablet 3dd1e318c2084e3a9a80987731920ac8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:28 UTC) from {username='slave'} at 127.0.0.1:33678
I20240503 13:56:28.886247 13427 tablet_service.cc:1467] Processing CreateTablet for tablet 0878ccb6135f4485b5ba35ae67a76d0c (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=18c0e326c0a144c382cd95a8c0052057]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:28.886484 13423 tablet_service.cc:1467] Processing CreateTablet for tablet 356ccd6fdf25470dba2a79367a37128e (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=18c0e326c0a144c382cd95a8c0052057]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:28.886554 13426 tablet_service.cc:1514] Processing DeleteTablet for tablet 101417485ac24fd9bb868fd81b2552e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:28 UTC) from {username='slave'} at 127.0.0.1:33678
I20240503 13:56:28.886713 13424 tablet_service.cc:1467] Processing CreateTablet for tablet c7d5e8c09ec14ac7891c8a63cb896878 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=18c0e326c0a144c382cd95a8c0052057]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:28.886502 13427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0878ccb6135f4485b5ba35ae67a76d0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:28.887545 13424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7d5e8c09ec14ac7891c8a63cb896878. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:28.889295 13033 tablet_service.cc:1467] Processing CreateTablet for tablet 0878ccb6135f4485b5ba35ae67a76d0c (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=18c0e326c0a144c382cd95a8c0052057]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:28.889557 13033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0878ccb6135f4485b5ba35ae67a76d0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:28.890071 13423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 356ccd6fdf25470dba2a79367a37128e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:28.891932 13164 tablet_service.cc:1514] Processing DeleteTablet for tablet 3dd1e318c2084e3a9a80987731920ac8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:28 UTC) from {username='slave'} at 127.0.0.1:41668
I20240503 13:56:28.892278 13165 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f79adb3e75c4a45998c73f29b65e312 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:28 UTC) from {username='slave'} at 127.0.0.1:41668
I20240503 13:56:28.892329 16089 tablet_bootstrap.cc:492] T c7d5e8c09ec14ac7891c8a63cb896878 P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:28.893216 16089 tablet_bootstrap.cc:654] T c7d5e8c09ec14ac7891c8a63cb896878 P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:28.897866 15472 tablet_service.cc:1467] Processing CreateTablet for tablet 356ccd6fdf25470dba2a79367a37128e (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=18c0e326c0a144c382cd95a8c0052057]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:28.898910 15472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 356ccd6fdf25470dba2a79367a37128e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:28.900163 15473 tablet_service.cc:1467] Processing CreateTablet for tablet c7d5e8c09ec14ac7891c8a63cb896878 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=18c0e326c0a144c382cd95a8c0052057]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:28.900421 15473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7d5e8c09ec14ac7891c8a63cb896878. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:28.902767 16089 tablet_bootstrap.cc:492] T c7d5e8c09ec14ac7891c8a63cb896878 P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:28.902858 16089 ts_tablet_manager.cc:1367] T c7d5e8c09ec14ac7891c8a63cb896878 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 13:56:28.903331 16089 raft_consensus.cc:348] T c7d5e8c09ec14ac7891c8a63cb896878 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:28.903432 16089 raft_consensus.cc:374] T c7d5e8c09ec14ac7891c8a63cb896878 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:28.903460 16089 raft_consensus.cc:729] T c7d5e8c09ec14ac7891c8a63cb896878 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:28.903087 15605 tablet_service.cc:1467] Processing CreateTablet for tablet c7d5e8c09ec14ac7891c8a63cb896878 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=18c0e326c0a144c382cd95a8c0052057]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:28.903577 16089 consensus_queue.cc:260] T c7d5e8c09ec14ac7891c8a63cb896878 P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:28.904063 15605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7d5e8c09ec14ac7891c8a63cb896878. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:28.904089 16089 ts_tablet_manager.cc:1398] T c7d5e8c09ec14ac7891c8a63cb896878 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:28.904278 16089 tablet_bootstrap.cc:492] T 0878ccb6135f4485b5ba35ae67a76d0c P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:28.920675 16092 tablet_replica.cc:330] T 3dd1e318c2084e3a9a80987731920ac8 P 8ccf5a4ac450414c9d3579bd5800b770: stopping tablet replica
I20240503 13:56:28.921550 16089 tablet_bootstrap.cc:654] T 0878ccb6135f4485b5ba35ae67a76d0c P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:28.923122 16092 raft_consensus.cc:2229] T 3dd1e318c2084e3a9a80987731920ac8 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:28.924237 16092 raft_consensus.cc:2258] T 3dd1e318c2084e3a9a80987731920ac8 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:28.928066 16089 tablet_bootstrap.cc:492] T 0878ccb6135f4485b5ba35ae67a76d0c P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:28.928169 16089 ts_tablet_manager.cc:1367] T 0878ccb6135f4485b5ba35ae67a76d0c P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.000s
I20240503 13:56:28.928635 16089 raft_consensus.cc:348] T 0878ccb6135f4485b5ba35ae67a76d0c P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:28.928766 16089 raft_consensus.cc:374] T 0878ccb6135f4485b5ba35ae67a76d0c P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:28.928804 16089 raft_consensus.cc:729] T 0878ccb6135f4485b5ba35ae67a76d0c P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:28.928923 16089 consensus_queue.cc:260] T 0878ccb6135f4485b5ba35ae67a76d0c P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:28.929200 16089 ts_tablet_manager.cc:1398] T 0878ccb6135f4485b5ba35ae67a76d0c P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:28.929363 16089 tablet_bootstrap.cc:492] T 356ccd6fdf25470dba2a79367a37128e P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:28.930285 16089 tablet_bootstrap.cc:654] T 356ccd6fdf25470dba2a79367a37128e P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:28.930940 15954 tablet_bootstrap.cc:492] T 356ccd6fdf25470dba2a79367a37128e P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:28.931064 15954 ts_tablet_manager.cc:1367] T 356ccd6fdf25470dba2a79367a37128e P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.046s user 0.002s sys 0.000s
I20240503 13:56:28.931217 16092 ts_tablet_manager.cc:1875] T 3dd1e318c2084e3a9a80987731920ac8 P 8ccf5a4ac450414c9d3579bd5800b770: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:28.931780 15954 raft_consensus.cc:348] T 356ccd6fdf25470dba2a79367a37128e P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:28.931891 15954 raft_consensus.cc:374] T 356ccd6fdf25470dba2a79367a37128e P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:28.931918 15954 raft_consensus.cc:729] T 356ccd6fdf25470dba2a79367a37128e P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:28.932055 15954 consensus_queue.cc:260] T 356ccd6fdf25470dba2a79367a37128e P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:28.932312 15954 ts_tablet_manager.cc:1398] T 356ccd6fdf25470dba2a79367a37128e P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:28.932504 15954 tablet_bootstrap.cc:492] T 0878ccb6135f4485b5ba35ae67a76d0c P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:28.934563 15954 tablet_bootstrap.cc:654] T 0878ccb6135f4485b5ba35ae67a76d0c P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:28.936534 16092 ts_tablet_manager.cc:1888] T 3dd1e318c2084e3a9a80987731920ac8 P 8ccf5a4ac450414c9d3579bd5800b770: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:56:28.937080 15954 tablet_bootstrap.cc:492] T 0878ccb6135f4485b5ba35ae67a76d0c P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:28.937161 15954 ts_tablet_manager.cc:1367] T 0878ccb6135f4485b5ba35ae67a76d0c P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:56:28.937594 15954 raft_consensus.cc:348] T 0878ccb6135f4485b5ba35ae67a76d0c P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:28.937688 15954 raft_consensus.cc:374] T 0878ccb6135f4485b5ba35ae67a76d0c P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:28.937713 15954 raft_consensus.cc:729] T 0878ccb6135f4485b5ba35ae67a76d0c P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:28.937819 15954 consensus_queue.cc:260] T 0878ccb6135f4485b5ba35ae67a76d0c P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:28.938041 15954 ts_tablet_manager.cc:1398] T 0878ccb6135f4485b5ba35ae67a76d0c P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:28.939327 16095 tablet_bootstrap.cc:492] T c7d5e8c09ec14ac7891c8a63cb896878 P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:28.942096 16089 tablet_bootstrap.cc:492] T 356ccd6fdf25470dba2a79367a37128e P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:28.942207 16089 ts_tablet_manager.cc:1367] T 356ccd6fdf25470dba2a79367a37128e P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 13:56:28.942680 16089 raft_consensus.cc:348] T 356ccd6fdf25470dba2a79367a37128e P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:28.942791 16089 raft_consensus.cc:374] T 356ccd6fdf25470dba2a79367a37128e P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:28.942831 16089 raft_consensus.cc:729] T 356ccd6fdf25470dba2a79367a37128e P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:28.942950 16089 consensus_queue.cc:260] T 356ccd6fdf25470dba2a79367a37128e P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:28.943241 16089 ts_tablet_manager.cc:1398] T 356ccd6fdf25470dba2a79367a37128e P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:28.944445 16092 log.cc:1196] T 3dd1e318c2084e3a9a80987731920ac8 P 8ccf5a4ac450414c9d3579bd5800b770: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/wal/wals/3dd1e318c2084e3a9a80987731920ac8
I20240503 13:56:28.944954 16092 ts_tablet_manager.cc:1909] T 3dd1e318c2084e3a9a80987731920ac8 P 8ccf5a4ac450414c9d3579bd5800b770: Deleting consensus metadata
I20240503 13:56:28.946537 12899 catalog_manager.cc:4830] TS 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387): tablet 3dd1e318c2084e3a9a80987731920ac8 (table rebalance_test_table_0 [id=81d7faaf4b654c90b42092653455de5d]) successfully deleted
I20240503 13:56:28.946805 16098 tablet_bootstrap.cc:492] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:28.947156 16092 tablet_replica.cc:330] T 5f79adb3e75c4a45998c73f29b65e312 P 8ccf5a4ac450414c9d3579bd5800b770: stopping tablet replica
I20240503 13:56:28.948899 16095 tablet_bootstrap.cc:654] T c7d5e8c09ec14ac7891c8a63cb896878 P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:28.949275 16098 tablet_bootstrap.cc:654] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:28.949491 16092 raft_consensus.cc:2229] T 5f79adb3e75c4a45998c73f29b65e312 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:28.949669 16092 raft_consensus.cc:2258] T 5f79adb3e75c4a45998c73f29b65e312 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:28.949842 16095 log.cc:824] T c7d5e8c09ec14ac7891c8a63cb896878 P 009f849ee7b54c80a3f5264695047e4d: Log is configured to *not* fsync() on all Append() calls
I20240503 13:56:28.950525 16092 ts_tablet_manager.cc:1875] T 5f79adb3e75c4a45998c73f29b65e312 P 8ccf5a4ac450414c9d3579bd5800b770: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:28.951174 15956 raft_consensus.cc:482] T 356ccd6fdf25470dba2a79367a37128e P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:28.951277 15956 raft_consensus.cc:504] T 356ccd6fdf25470dba2a79367a37128e P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:28.952286 15956 leader_election.cc:288] T 356ccd6fdf25470dba2a79367a37128e P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153)
I20240503 13:56:28.953617 16098 log.cc:824] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546: Log is configured to *not* fsync() on all Append() calls
I20240503 13:56:28.954582 16092 ts_tablet_manager.cc:1888] T 5f79adb3e75c4a45998c73f29b65e312 P 8ccf5a4ac450414c9d3579bd5800b770: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:56:28.954674 16092 log.cc:1196] T 5f79adb3e75c4a45998c73f29b65e312 P 8ccf5a4ac450414c9d3579bd5800b770: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/wal/wals/5f79adb3e75c4a45998c73f29b65e312
I20240503 13:56:28.955888 16098 tablet_bootstrap.cc:492] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546: No bootstrap required, opened a new log
I20240503 13:56:28.956081 16098 ts_tablet_manager.cc:1367] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240503 13:56:28.959923 16098 raft_consensus.cc:348] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:28.960186 16098 raft_consensus.cc:374] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:28.960230 16098 raft_consensus.cc:729] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:28.960711 16086 tablet_replica.cc:330] T 5f79adb3e75c4a45998c73f29b65e312 P ba0194098aa84449ae17e8a0d4c43703: stopping tablet replica
I20240503 13:56:28.960745 16098 consensus_queue.cc:260] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:28.960922 16092 ts_tablet_manager.cc:1909] T 5f79adb3e75c4a45998c73f29b65e312 P 8ccf5a4ac450414c9d3579bd5800b770: Deleting consensus metadata
I20240503 13:56:28.961045 16086 raft_consensus.cc:2229] T 5f79adb3e75c4a45998c73f29b65e312 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:28.961248 16086 raft_consensus.cc:2258] T 5f79adb3e75c4a45998c73f29b65e312 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:28.961755 12899 catalog_manager.cc:4830] TS 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387): tablet 5f79adb3e75c4a45998c73f29b65e312 (table rebalance_test_table_0 [id=81d7faaf4b654c90b42092653455de5d]) successfully deleted
I20240503 13:56:28.961894 16098 ts_tablet_manager.cc:1398] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20240503 13:56:28.962182 16086 ts_tablet_manager.cc:1875] T 5f79adb3e75c4a45998c73f29b65e312 P ba0194098aa84449ae17e8a0d4c43703: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:28.962218 16098 tablet_bootstrap.cc:492] T 356ccd6fdf25470dba2a79367a37128e P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:28.963222 16098 tablet_bootstrap.cc:654] T 356ccd6fdf25470dba2a79367a37128e P 61b887f0d40d497badeca5604c1f0546: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:28.964277 16098 tablet_bootstrap.cc:492] T 356ccd6fdf25470dba2a79367a37128e P 61b887f0d40d497badeca5604c1f0546: No bootstrap required, opened a new log
I20240503 13:56:28.964366 16098 ts_tablet_manager.cc:1367] T 356ccd6fdf25470dba2a79367a37128e P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:56:28.964741 16085 tablet_replica.cc:330] T 3dd1e318c2084e3a9a80987731920ac8 P 60ba3ffe3b7746469834cd91c84be14e: stopping tablet replica
I20240503 13:56:28.964956 16098 raft_consensus.cc:348] T 356ccd6fdf25470dba2a79367a37128e P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:28.965032 16085 raft_consensus.cc:2229] T 3dd1e318c2084e3a9a80987731920ac8 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:28.965046 16098 raft_consensus.cc:374] T 356ccd6fdf25470dba2a79367a37128e P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:28.965085 16098 raft_consensus.cc:729] T 356ccd6fdf25470dba2a79367a37128e P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:28.965209 16085 raft_consensus.cc:2258] T 3dd1e318c2084e3a9a80987731920ac8 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:28.965198 16098 consensus_queue.cc:260] T 356ccd6fdf25470dba2a79367a37128e P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:28.965425 16098 ts_tablet_manager.cc:1398] T 356ccd6fdf25470dba2a79367a37128e P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:28.966084 16085 ts_tablet_manager.cc:1875] T 3dd1e318c2084e3a9a80987731920ac8 P 60ba3ffe3b7746469834cd91c84be14e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:28.968120 16085 ts_tablet_manager.cc:1888] T 3dd1e318c2084e3a9a80987731920ac8 P 60ba3ffe3b7746469834cd91c84be14e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:56:28.968204 16085 log.cc:1196] T 3dd1e318c2084e3a9a80987731920ac8 P 60ba3ffe3b7746469834cd91c84be14e: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal/wals/3dd1e318c2084e3a9a80987731920ac8
I20240503 13:56:28.968436 16085 ts_tablet_manager.cc:1909] T 3dd1e318c2084e3a9a80987731920ac8 P 60ba3ffe3b7746469834cd91c84be14e: Deleting consensus metadata
I20240503 13:56:28.968788 16087 tablet_replica.cc:330] T 3dd1e318c2084e3a9a80987731920ac8 P 7690ee79577c45e3a2c8f8b552c0bff8: stopping tablet replica
I20240503 13:56:28.969097 16087 raft_consensus.cc:2229] T 3dd1e318c2084e3a9a80987731920ac8 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:56:28.969149 15295 raft_consensus.cc:482] T 356ccd6fdf25470dba2a79367a37128e P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:28.969240 15295 raft_consensus.cc:504] T 356ccd6fdf25470dba2a79367a37128e P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:28.969398 16087 raft_consensus.cc:2258] T 3dd1e318c2084e3a9a80987731920ac8 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:28.969689 15295 leader_election.cc:288] T 356ccd6fdf25470dba2a79367a37128e P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153), 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885)
I20240503 13:56:28.970532 16087 ts_tablet_manager.cc:1875] T 3dd1e318c2084e3a9a80987731920ac8 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:28.972730 16090 tablet_bootstrap.cc:492] T 0878ccb6135f4485b5ba35ae67a76d0c P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:28.972718 16087 ts_tablet_manager.cc:1888] T 3dd1e318c2084e3a9a80987731920ac8 P 7690ee79577c45e3a2c8f8b552c0bff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:56:28.972800 16087 log.cc:1196] T 3dd1e318c2084e3a9a80987731920ac8 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal/wals/3dd1e318c2084e3a9a80987731920ac8
I20240503 13:56:28.973088 16087 ts_tablet_manager.cc:1909] T 3dd1e318c2084e3a9a80987731920ac8 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting consensus metadata
I20240503 13:56:28.973706 16090 tablet_bootstrap.cc:654] T 0878ccb6135f4485b5ba35ae67a76d0c P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:28.973726 12897 catalog_manager.cc:4830] TS 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695): tablet 3dd1e318c2084e3a9a80987731920ac8 (table rebalance_test_table_0 [id=81d7faaf4b654c90b42092653455de5d]) successfully deleted
I20240503 13:56:28.974115 16087 tablet_replica.cc:330] T 101417485ac24fd9bb868fd81b2552e5 P 7690ee79577c45e3a2c8f8b552c0bff8: stopping tablet replica
I20240503 13:56:28.974221 12896 catalog_manager.cc:4830] TS 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109): tablet 3dd1e318c2084e3a9a80987731920ac8 (table rebalance_test_table_0 [id=81d7faaf4b654c90b42092653455de5d]) successfully deleted
I20240503 13:56:28.974268 16087 raft_consensus.cc:2229] T 101417485ac24fd9bb868fd81b2552e5 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:56:28.974018 16085 tablet_replica.cc:330] T 5f79adb3e75c4a45998c73f29b65e312 P 60ba3ffe3b7746469834cd91c84be14e: stopping tablet replica
I20240503 13:56:28.974546 16085 raft_consensus.cc:2229] T 5f79adb3e75c4a45998c73f29b65e312 P 60ba3ffe3b7746469834cd91c84be14e [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:56:28.974550 16087 raft_consensus.cc:2258] T 101417485ac24fd9bb868fd81b2552e5 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:28.974800 16085 raft_consensus.cc:2258] T 5f79adb3e75c4a45998c73f29b65e312 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:28.975181 16087 ts_tablet_manager.cc:1875] T 101417485ac24fd9bb868fd81b2552e5 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:28.975262 15493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "356ccd6fdf25470dba2a79367a37128e" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546" is_pre_election: true
I20240503 13:56:28.975545 15493 raft_consensus.cc:2454] T 356ccd6fdf25470dba2a79367a37128e P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 0.
I20240503 13:56:28.975967 13383 leader_election.cc:302] T 356ccd6fdf25470dba2a79367a37128e P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61b887f0d40d497badeca5604c1f0546, 7690ee79577c45e3a2c8f8b552c0bff8; no voters:
I20240503 13:56:28.976131 15295 raft_consensus.cc:2781] T 356ccd6fdf25470dba2a79367a37128e P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:28.976178 15295 raft_consensus.cc:482] T 356ccd6fdf25470dba2a79367a37128e P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:28.976210 15295 raft_consensus.cc:3037] T 356ccd6fdf25470dba2a79367a37128e P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:28.976615 16085 ts_tablet_manager.cc:1875] T 5f79adb3e75c4a45998c73f29b65e312 P 60ba3ffe3b7746469834cd91c84be14e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:28.977134 15295 raft_consensus.cc:504] T 356ccd6fdf25470dba2a79367a37128e P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:28.977236 16087 ts_tablet_manager.cc:1888] T 101417485ac24fd9bb868fd81b2552e5 P 7690ee79577c45e3a2c8f8b552c0bff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:56:28.977301 16087 log.cc:1196] T 101417485ac24fd9bb868fd81b2552e5 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal/wals/101417485ac24fd9bb868fd81b2552e5
I20240503 13:56:28.977488 15295 leader_election.cc:288] T 356ccd6fdf25470dba2a79367a37128e P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 election: Requested vote from peers 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153), 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885)
I20240503 13:56:28.977551 16087 ts_tablet_manager.cc:1909] T 101417485ac24fd9bb868fd81b2552e5 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting consensus metadata
I20240503 13:56:28.978602 16085 ts_tablet_manager.cc:1888] T 5f79adb3e75c4a45998c73f29b65e312 P 60ba3ffe3b7746469834cd91c84be14e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:56:28.978588 12896 catalog_manager.cc:4830] TS 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109): tablet 101417485ac24fd9bb868fd81b2552e5 (table rebalance_test_table_0 [id=81d7faaf4b654c90b42092653455de5d]) successfully deleted
I20240503 13:56:28.978698 16085 log.cc:1196] T 5f79adb3e75c4a45998c73f29b65e312 P 60ba3ffe3b7746469834cd91c84be14e: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal/wals/5f79adb3e75c4a45998c73f29b65e312
I20240503 13:56:28.978730 16090 tablet_bootstrap.cc:492] T 0878ccb6135f4485b5ba35ae67a76d0c P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:28.978857 16090 ts_tablet_manager.cc:1367] T 0878ccb6135f4485b5ba35ae67a76d0c P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 13:56:28.978938 16085 ts_tablet_manager.cc:1909] T 5f79adb3e75c4a45998c73f29b65e312 P 60ba3ffe3b7746469834cd91c84be14e: Deleting consensus metadata
I20240503 13:56:28.979286 16090 raft_consensus.cc:348] T 0878ccb6135f4485b5ba35ae67a76d0c P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:28.979416 16090 raft_consensus.cc:374] T 0878ccb6135f4485b5ba35ae67a76d0c P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:28.979467 16090 raft_consensus.cc:729] T 0878ccb6135f4485b5ba35ae67a76d0c P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:28.979429 15493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "356ccd6fdf25470dba2a79367a37128e" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546"
I20240503 13:56:28.979553 12897 catalog_manager.cc:4830] TS 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695): tablet 5f79adb3e75c4a45998c73f29b65e312 (table rebalance_test_table_0 [id=81d7faaf4b654c90b42092653455de5d]) successfully deleted
I20240503 13:56:28.979593 15493 raft_consensus.cc:3037] T 356ccd6fdf25470dba2a79367a37128e P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:28.979651 16090 consensus_queue.cc:260] T 0878ccb6135f4485b5ba35ae67a76d0c P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:28.979787 16085 tablet_replica.cc:330] T 101417485ac24fd9bb868fd81b2552e5 P 60ba3ffe3b7746469834cd91c84be14e: stopping tablet replica
I20240503 13:56:28.979945 16085 raft_consensus.cc:2229] T 101417485ac24fd9bb868fd81b2552e5 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:28.980094 16085 raft_consensus.cc:2258] T 101417485ac24fd9bb868fd81b2552e5 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:28.980589 16085 ts_tablet_manager.cc:1875] T 101417485ac24fd9bb868fd81b2552e5 P 60ba3ffe3b7746469834cd91c84be14e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:28.980635 15493 raft_consensus.cc:2454] T 356ccd6fdf25470dba2a79367a37128e P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 1.
I20240503 13:56:28.980903 16090 ts_tablet_manager.cc:1398] T 0878ccb6135f4485b5ba35ae67a76d0c P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:28.981097 13383 leader_election.cc:302] T 356ccd6fdf25470dba2a79367a37128e P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61b887f0d40d497badeca5604c1f0546, 7690ee79577c45e3a2c8f8b552c0bff8; no voters:
I20240503 13:56:28.981161 16086 ts_tablet_manager.cc:1888] T 5f79adb3e75c4a45998c73f29b65e312 P ba0194098aa84449ae17e8a0d4c43703: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:56:28.981313 16086 log.cc:1196] T 5f79adb3e75c4a45998c73f29b65e312 P ba0194098aa84449ae17e8a0d4c43703: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/wal/wals/5f79adb3e75c4a45998c73f29b65e312
I20240503 13:56:28.981644 16086 ts_tablet_manager.cc:1909] T 5f79adb3e75c4a45998c73f29b65e312 P ba0194098aa84449ae17e8a0d4c43703: Deleting consensus metadata
I20240503 13:56:28.982551 15295 raft_consensus.cc:2781] T 356ccd6fdf25470dba2a79367a37128e P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:28.982638 15295 raft_consensus.cc:686] T 356ccd6fdf25470dba2a79367a37128e P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 LEADER]: Becoming Leader. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Running, Role: LEADER
I20240503 13:56:28.982652 16085 ts_tablet_manager.cc:1888] T 101417485ac24fd9bb868fd81b2552e5 P 60ba3ffe3b7746469834cd91c84be14e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:56:28.982704 16085 log.cc:1196] T 101417485ac24fd9bb868fd81b2552e5 P 60ba3ffe3b7746469834cd91c84be14e: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal/wals/101417485ac24fd9bb868fd81b2552e5
I20240503 13:56:28.982767 15295 consensus_queue.cc:237] T 356ccd6fdf25470dba2a79367a37128e P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:28.982915 16085 ts_tablet_manager.cc:1909] T 101417485ac24fd9bb868fd81b2552e5 P 60ba3ffe3b7746469834cd91c84be14e: Deleting consensus metadata
I20240503 13:56:28.983597 12897 catalog_manager.cc:4830] TS 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695): tablet 101417485ac24fd9bb868fd81b2552e5 (table rebalance_test_table_0 [id=81d7faaf4b654c90b42092653455de5d]) successfully deleted
I20240503 13:56:28.988042 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "356ccd6fdf25470dba2a79367a37128e" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2" is_pre_election: true
I20240503 13:56:28.988241 15757 raft_consensus.cc:2454] T 356ccd6fdf25470dba2a79367a37128e P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 0.
I20240503 13:56:28.988260 12899 catalog_manager.cc:4830] TS ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279): tablet 5f79adb3e75c4a45998c73f29b65e312 (table rebalance_test_table_0 [id=81d7faaf4b654c90b42092653455de5d]) successfully deleted
I20240503 13:56:28.988478 15758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "356ccd6fdf25470dba2a79367a37128e" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2"
I20240503 13:56:28.988605 15758 raft_consensus.cc:3037] T 356ccd6fdf25470dba2a79367a37128e P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:28.988972 16086 tablet_replica.cc:330] T 101417485ac24fd9bb868fd81b2552e5 P ba0194098aa84449ae17e8a0d4c43703: stopping tablet replica
I20240503 13:56:28.989538 15758 raft_consensus.cc:2454] T 356ccd6fdf25470dba2a79367a37128e P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 1.
I20240503 13:56:28.989663 16086 raft_consensus.cc:2229] T 101417485ac24fd9bb868fd81b2552e5 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:28.989810 16086 raft_consensus.cc:2258] T 101417485ac24fd9bb868fd81b2552e5 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:28.990254 12905 catalog_manager.cc:5484] T 356ccd6fdf25470dba2a79367a37128e P 7690ee79577c45e3a2c8f8b552c0bff8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68). New cstate: current_term: 1 leader_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:28.990399 16086 ts_tablet_manager.cc:1875] T 101417485ac24fd9bb868fd81b2552e5 P ba0194098aa84449ae17e8a0d4c43703: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:28.994410 15671 heartbeater.cc:503] Master 127.6.98.126:44613 was elected leader, sending a full tablet report...
I20240503 13:56:28.996646 15493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "356ccd6fdf25470dba2a79367a37128e" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546" is_pre_election: true
I20240503 13:56:28.996861 15493 raft_consensus.cc:2379] T 356ccd6fdf25470dba2a79367a37128e P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 76c386bef61a4bd5a907266a5118ffe2 in current term 1: Already voted for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in this term.
I20240503 13:56:28.998536 12905 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:28.998662 12905 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:28.999644 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "356ccd6fdf25470dba2a79367a37128e" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
I20240503 13:56:29.000068 15691 leader_election.cc:302] T 356ccd6fdf25470dba2a79367a37128e P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76c386bef61a4bd5a907266a5118ffe2; no voters: 61b887f0d40d497badeca5604c1f0546, 7690ee79577c45e3a2c8f8b552c0bff8
I20240503 13:56:29.000260 16095 tablet_bootstrap.cc:492] T c7d5e8c09ec14ac7891c8a63cb896878 P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:29.000556 15956 raft_consensus.cc:2726] T 356ccd6fdf25470dba2a79367a37128e P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 13:56:29.000458 16095 ts_tablet_manager.cc:1367] T c7d5e8c09ec14ac7891c8a63cb896878 P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.061s user 0.004s sys 0.000s
I20240503 13:56:28.996438 16086 ts_tablet_manager.cc:1888] T 101417485ac24fd9bb868fd81b2552e5 P ba0194098aa84449ae17e8a0d4c43703: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:56:29.000912 16086 log.cc:1196] T 101417485ac24fd9bb868fd81b2552e5 P ba0194098aa84449ae17e8a0d4c43703: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/wal/wals/101417485ac24fd9bb868fd81b2552e5
I20240503 13:56:29.001232 12905 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Sending DeleteTablet for 3 replicas of tablet 4ab2e8e604954be39ae6b8f1e028c4ba
I20240503 13:56:29.001259 16086 ts_tablet_manager.cc:1909] T 101417485ac24fd9bb868fd81b2552e5 P ba0194098aa84449ae17e8a0d4c43703: Deleting consensus metadata
I20240503 13:56:29.002286 15738 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ab2e8e604954be39ae6b8f1e028c4ba with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:28 UTC) from {username='slave'} at 127.0.0.1:44984
I20240503 13:56:29.002697 13033 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ab2e8e604954be39ae6b8f1e028c4ba with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:28 UTC) from {username='slave'} at 127.0.0.1:59458
I20240503 13:56:29.003841 12899 catalog_manager.cc:4830] TS ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279): tablet 101417485ac24fd9bb868fd81b2552e5 (table rebalance_test_table_0 [id=81d7faaf4b654c90b42092653455de5d]) successfully deleted
I20240503 13:56:29.003846 13295 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ab2e8e604954be39ae6b8f1e028c4ba with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:28 UTC) from {username='slave'} at 127.0.0.1:33250
I20240503 13:56:29.004052 16086 tablet_replica.cc:330] T 4ab2e8e604954be39ae6b8f1e028c4ba P ba0194098aa84449ae17e8a0d4c43703: stopping tablet replica
I20240503 13:56:29.004237 16086 raft_consensus.cc:2229] T 4ab2e8e604954be39ae6b8f1e028c4ba P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:29.004401 16086 raft_consensus.cc:2258] T 4ab2e8e604954be39ae6b8f1e028c4ba P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:29.005189 16086 ts_tablet_manager.cc:1875] T 4ab2e8e604954be39ae6b8f1e028c4ba P ba0194098aa84449ae17e8a0d4c43703: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:29.005111 16095 raft_consensus.cc:348] T c7d5e8c09ec14ac7891c8a63cb896878 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:29.005388 16095 raft_consensus.cc:374] T c7d5e8c09ec14ac7891c8a63cb896878 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:29.005475 16095 raft_consensus.cc:729] T c7d5e8c09ec14ac7891c8a63cb896878 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:29.006016 16095 consensus_queue.cc:260] T c7d5e8c09ec14ac7891c8a63cb896878 P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:29.006958 16095 ts_tablet_manager.cc:1398] T c7d5e8c09ec14ac7891c8a63cb896878 P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240503 13:56:29.007870 16086 ts_tablet_manager.cc:1888] T 4ab2e8e604954be39ae6b8f1e028c4ba P ba0194098aa84449ae17e8a0d4c43703: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:29.008000 16086 log.cc:1196] T 4ab2e8e604954be39ae6b8f1e028c4ba P ba0194098aa84449ae17e8a0d4c43703: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/wal/wals/4ab2e8e604954be39ae6b8f1e028c4ba
I20240503 13:56:29.008302 16086 ts_tablet_manager.cc:1909] T 4ab2e8e604954be39ae6b8f1e028c4ba P ba0194098aa84449ae17e8a0d4c43703: Deleting consensus metadata
I20240503 13:56:29.008308 16110 tablet_replica.cc:330] T 4ab2e8e604954be39ae6b8f1e028c4ba P 76c386bef61a4bd5a907266a5118ffe2: stopping tablet replica
I20240503 13:56:29.008641 16110 raft_consensus.cc:2229] T 4ab2e8e604954be39ae6b8f1e028c4ba P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:29.009163 12899 catalog_manager.cc:4830] TS ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279): tablet 4ab2e8e604954be39ae6b8f1e028c4ba (table rebalancer_dynamic_table_new_name [id=d6641e092440471ba33512c2cb1d2d2b]) successfully deleted
I20240503 13:56:29.008972 16085 tablet_replica.cc:330] T 4ab2e8e604954be39ae6b8f1e028c4ba P 60ba3ffe3b7746469834cd91c84be14e: stopping tablet replica
I20240503 13:56:29.009486 16085 raft_consensus.cc:2229] T 4ab2e8e604954be39ae6b8f1e028c4ba P 60ba3ffe3b7746469834cd91c84be14e [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:56:29.009694 16085 raft_consensus.cc:2258] T 4ab2e8e604954be39ae6b8f1e028c4ba P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:29.010310 16085 ts_tablet_manager.cc:1875] T 4ab2e8e604954be39ae6b8f1e028c4ba P 60ba3ffe3b7746469834cd91c84be14e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:29.012295 16085 ts_tablet_manager.cc:1888] T 4ab2e8e604954be39ae6b8f1e028c4ba P 60ba3ffe3b7746469834cd91c84be14e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:29.012364 16085 log.cc:1196] T 4ab2e8e604954be39ae6b8f1e028c4ba P 60ba3ffe3b7746469834cd91c84be14e: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal/wals/4ab2e8e604954be39ae6b8f1e028c4ba
I20240503 13:56:29.012658 16085 ts_tablet_manager.cc:1909] T 4ab2e8e604954be39ae6b8f1e028c4ba P 60ba3ffe3b7746469834cd91c84be14e: Deleting consensus metadata
I20240503 13:56:29.012755 16110 raft_consensus.cc:2258] T 4ab2e8e604954be39ae6b8f1e028c4ba P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:29.013326 12897 catalog_manager.cc:4830] TS 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695): tablet 4ab2e8e604954be39ae6b8f1e028c4ba (table rebalancer_dynamic_table_new_name [id=d6641e092440471ba33512c2cb1d2d2b]) successfully deleted
I20240503 13:56:29.014297 16110 ts_tablet_manager.cc:1875] T 4ab2e8e604954be39ae6b8f1e028c4ba P 76c386bef61a4bd5a907266a5118ffe2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:29.016691 16103 raft_consensus.cc:482] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:29.016796 16103 raft_consensus.cc:504] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:29.017061 16110 ts_tablet_manager.cc:1888] T 4ab2e8e604954be39ae6b8f1e028c4ba P 76c386bef61a4bd5a907266a5118ffe2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:29.017171 16110 log.cc:1196] T 4ab2e8e604954be39ae6b8f1e028c4ba P 76c386bef61a4bd5a907266a5118ffe2: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/wal/wals/4ab2e8e604954be39ae6b8f1e028c4ba
I20240503 13:56:29.017541 16110 ts_tablet_manager.cc:1909] T 4ab2e8e604954be39ae6b8f1e028c4ba P 76c386bef61a4bd5a907266a5118ffe2: Deleting consensus metadata
I20240503 13:56:29.017848 16103 leader_election.cc:288] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951)
I20240503 13:56:29.019430 12899 catalog_manager.cc:4830] TS 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885): tablet 4ab2e8e604954be39ae6b8f1e028c4ba (table rebalancer_dynamic_table_new_name [id=d6641e092440471ba33512c2cb1d2d2b]) successfully deleted
I20240503 13:56:29.022974 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7d5e8c09ec14ac7891c8a63cb896878" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
I20240503 13:56:29.023144 13447 raft_consensus.cc:2454] T c7d5e8c09ec14ac7891c8a63cb896878 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 0.
I20240503 13:56:29.023559 15625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7d5e8c09ec14ac7891c8a63cb896878" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d" is_pre_election: true
I20240503 13:56:29.023851 15625 raft_consensus.cc:2454] T c7d5e8c09ec14ac7891c8a63cb896878 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 0.
I20240503 13:56:29.024194 15426 leader_election.cc:302] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61b887f0d40d497badeca5604c1f0546, 7690ee79577c45e3a2c8f8b552c0bff8; no voters:
I20240503 13:56:29.024467 16103 raft_consensus.cc:2781] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:29.024571 16103 raft_consensus.cc:482] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:29.024629 16103 raft_consensus.cc:3037] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:29.025764 16103 raft_consensus.cc:504] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:29.026168 16103 leader_election.cc:288] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 election: Requested vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951)
I20240503 13:56:29.026588 15625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7d5e8c09ec14ac7891c8a63cb896878" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d"
I20240503 13:56:29.026607 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7d5e8c09ec14ac7891c8a63cb896878" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
I20240503 13:56:29.026731 13447 raft_consensus.cc:3037] T c7d5e8c09ec14ac7891c8a63cb896878 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:29.026715 15625 raft_consensus.cc:3037] T c7d5e8c09ec14ac7891c8a63cb896878 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:29.027794 13447 raft_consensus.cc:2454] T c7d5e8c09ec14ac7891c8a63cb896878 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 1.
I20240503 13:56:29.028187 15625 raft_consensus.cc:2454] T c7d5e8c09ec14ac7891c8a63cb896878 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 1.
I20240503 13:56:29.029031 15426 leader_election.cc:302] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 61b887f0d40d497badeca5604c1f0546; no voters:
I20240503 13:56:29.029238 16103 raft_consensus.cc:2781] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:29.029472 16103 raft_consensus.cc:686] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 [term 1 LEADER]: Becoming Leader. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Running, Role: LEADER
I20240503 13:56:29.029894 16103 consensus_queue.cc:237] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:29.033874 12906 catalog_manager.cc:5484] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61b887f0d40d497badeca5604c1f0546 (127.6.98.69). New cstate: current_term: 1 leader_uuid: "61b887f0d40d497badeca5604c1f0546" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
W20240503 13:56:29.039948 15540 tablet.cc:2369] T 356ccd6fdf25470dba2a79367a37128e P 61b887f0d40d497badeca5604c1f0546: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 13:56:29.104535 12906 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:56:29.110960 15605 tablet_service.cc:1467] Processing CreateTablet for tablet 97cd45c1b3b342a5aa1e3b1d24d89963 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4303f2a0523d40c090cfaaa50a92e519]), partition=
I20240503 13:56:29.111263 15605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97cd45c1b3b342a5aa1e3b1d24d89963. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:29.112390 15738 tablet_service.cc:1467] Processing CreateTablet for tablet 97cd45c1b3b342a5aa1e3b1d24d89963 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4303f2a0523d40c090cfaaa50a92e519]), partition=
I20240503 13:56:29.112962 15738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97cd45c1b3b342a5aa1e3b1d24d89963. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:29.113904 16095 tablet_bootstrap.cc:492] T 97cd45c1b3b342a5aa1e3b1d24d89963 P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:29.114840 16095 tablet_bootstrap.cc:654] T 97cd45c1b3b342a5aa1e3b1d24d89963 P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:29.115064 15954 tablet_bootstrap.cc:492] T 97cd45c1b3b342a5aa1e3b1d24d89963 P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:29.115953 15954 tablet_bootstrap.cc:654] T 97cd45c1b3b342a5aa1e3b1d24d89963 P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:29.122116 15869 tablet_service.cc:1467] Processing CreateTablet for tablet 97cd45c1b3b342a5aa1e3b1d24d89963 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4303f2a0523d40c090cfaaa50a92e519]), partition=
I20240503 13:56:29.122874 15869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97cd45c1b3b342a5aa1e3b1d24d89963. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:29.123859 15954 tablet_bootstrap.cc:492] T 97cd45c1b3b342a5aa1e3b1d24d89963 P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:29.123984 15954 ts_tablet_manager.cc:1367] T 97cd45c1b3b342a5aa1e3b1d24d89963 P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 13:56:29.124536 15954 raft_consensus.cc:348] T 97cd45c1b3b342a5aa1e3b1d24d89963 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:29.124686 15954 raft_consensus.cc:374] T 97cd45c1b3b342a5aa1e3b1d24d89963 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:29.124723 15954 raft_consensus.cc:729] T 97cd45c1b3b342a5aa1e3b1d24d89963 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:29.124840 15954 consensus_queue.cc:260] T 97cd45c1b3b342a5aa1e3b1d24d89963 P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:29.125118 15954 ts_tablet_manager.cc:1398] T 97cd45c1b3b342a5aa1e3b1d24d89963 P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:29.133834 16095 tablet_bootstrap.cc:492] T 97cd45c1b3b342a5aa1e3b1d24d89963 P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:29.133965 16095 ts_tablet_manager.cc:1367] T 97cd45c1b3b342a5aa1e3b1d24d89963 P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20240503 13:56:29.134616 16095 raft_consensus.cc:348] T 97cd45c1b3b342a5aa1e3b1d24d89963 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:29.134732 16095 raft_consensus.cc:374] T 97cd45c1b3b342a5aa1e3b1d24d89963 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:29.134768 16095 raft_consensus.cc:729] T 97cd45c1b3b342a5aa1e3b1d24d89963 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:29.134886 16095 consensus_queue.cc:260] T 97cd45c1b3b342a5aa1e3b1d24d89963 P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:29.135172 16095 ts_tablet_manager.cc:1398] T 97cd45c1b3b342a5aa1e3b1d24d89963 P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:29.139067 16159 tablet_bootstrap.cc:492] T 97cd45c1b3b342a5aa1e3b1d24d89963 P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:29.142659 16159 tablet_bootstrap.cc:654] T 97cd45c1b3b342a5aa1e3b1d24d89963 P e47105a0f46e4511b7112e571688a4c7: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:29.151341 16159 log.cc:824] T 97cd45c1b3b342a5aa1e3b1d24d89963 P e47105a0f46e4511b7112e571688a4c7: Log is configured to *not* fsync() on all Append() calls
I20240503 13:56:29.153621 16159 tablet_bootstrap.cc:492] T 97cd45c1b3b342a5aa1e3b1d24d89963 P e47105a0f46e4511b7112e571688a4c7: No bootstrap required, opened a new log
I20240503 13:56:29.153930 16159 ts_tablet_manager.cc:1367] T 97cd45c1b3b342a5aa1e3b1d24d89963 P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.001s
I20240503 13:56:29.155880 15956 raft_consensus.cc:482] T 0878ccb6135f4485b5ba35ae67a76d0c P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:29.155997 15956 raft_consensus.cc:504] T 0878ccb6135f4485b5ba35ae67a76d0c P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:29.156563 15956 leader_election.cc:288] T 0878ccb6135f4485b5ba35ae67a76d0c P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:29.156842 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0878ccb6135f4485b5ba35ae67a76d0c" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
I20240503 13:56:29.156988 13446 raft_consensus.cc:2454] T 0878ccb6135f4485b5ba35ae67a76d0c P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 0.
I20240503 13:56:29.157898 15691 leader_election.cc:302] T 0878ccb6135f4485b5ba35ae67a76d0c P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, 76c386bef61a4bd5a907266a5118ffe2; no voters:
I20240503 13:56:29.158052 16159 raft_consensus.cc:348] T 97cd45c1b3b342a5aa1e3b1d24d89963 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:29.158397 16159 raft_consensus.cc:374] T 97cd45c1b3b342a5aa1e3b1d24d89963 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:29.158495 16159 raft_consensus.cc:729] T 97cd45c1b3b342a5aa1e3b1d24d89963 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:29.159116 16159 consensus_queue.cc:260] T 97cd45c1b3b342a5aa1e3b1d24d89963 P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:29.159624 15956 raft_consensus.cc:2781] T 0878ccb6135f4485b5ba35ae67a76d0c P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:29.159690 15956 raft_consensus.cc:482] T 0878ccb6135f4485b5ba35ae67a76d0c P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:29.159727 15956 raft_consensus.cc:3037] T 0878ccb6135f4485b5ba35ae67a76d0c P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:29.160077 16159 ts_tablet_manager.cc:1398] T 97cd45c1b3b342a5aa1e3b1d24d89963 P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20240503 13:56:29.160266 15935 heartbeater.cc:503] Master 127.6.98.126:44613 was elected leader, sending a full tablet report...
I20240503 13:56:29.160866 15956 raft_consensus.cc:504] T 0878ccb6135f4485b5ba35ae67a76d0c P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:29.161233 15956 leader_election.cc:288] T 0878ccb6135f4485b5ba35ae67a76d0c P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 election: Requested vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:29.161736 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0878ccb6135f4485b5ba35ae67a76d0c" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
I20240503 13:56:29.161854 13446 raft_consensus.cc:3037] T 0878ccb6135f4485b5ba35ae67a76d0c P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:29.162715 13446 raft_consensus.cc:2454] T 0878ccb6135f4485b5ba35ae67a76d0c P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 1.
I20240503 13:56:29.163462 15691 leader_election.cc:302] T 0878ccb6135f4485b5ba35ae67a76d0c P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, 76c386bef61a4bd5a907266a5118ffe2; no voters:
I20240503 13:56:29.165107 13051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0878ccb6135f4485b5ba35ae67a76d0c" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e"
I20240503 13:56:29.165241 13051 raft_consensus.cc:3037] T 0878ccb6135f4485b5ba35ae67a76d0c P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:29.165776 13052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0878ccb6135f4485b5ba35ae67a76d0c" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e" is_pre_election: true
I20240503 13:56:29.166260 13051 raft_consensus.cc:2454] T 0878ccb6135f4485b5ba35ae67a76d0c P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 1.
I20240503 13:56:29.166304 15956 raft_consensus.cc:2781] T 0878ccb6135f4485b5ba35ae67a76d0c P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:29.166492 15956 raft_consensus.cc:686] T 0878ccb6135f4485b5ba35ae67a76d0c P 76c386bef61a4bd5a907266a5118ffe2 [term 1 LEADER]: Becoming Leader. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Running, Role: LEADER
I20240503 13:56:29.166728 15956 consensus_queue.cc:237] T 0878ccb6135f4485b5ba35ae67a76d0c P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:29.168905 12905 catalog_manager.cc:5484] T 0878ccb6135f4485b5ba35ae67a76d0c P 76c386bef61a4bd5a907266a5118ffe2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71). New cstate: current_term: 1 leader_uuid: "76c386bef61a4bd5a907266a5118ffe2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
W20240503 13:56:29.230930 15672 tablet.cc:2369] T 97cd45c1b3b342a5aa1e3b1d24d89963 P 009f849ee7b54c80a3f5264695047e4d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 13:56:29.268169 16162 raft_consensus.cc:482] T 97cd45c1b3b342a5aa1e3b1d24d89963 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:29.268309 16162 raft_consensus.cc:504] T 97cd45c1b3b342a5aa1e3b1d24d89963 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:29.268850 16162 leader_election.cc:288] T 97cd45c1b3b342a5aa1e3b1d24d89963 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:29.272029 15625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97cd45c1b3b342a5aa1e3b1d24d89963" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d" is_pre_election: true
I20240503 13:56:29.272167 15625 raft_consensus.cc:2454] T 97cd45c1b3b342a5aa1e3b1d24d89963 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 0.
I20240503 13:56:29.272370 15889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97cd45c1b3b342a5aa1e3b1d24d89963" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7" is_pre_election: true
I20240503 13:56:29.272446 15691 leader_election.cc:302] T 97cd45c1b3b342a5aa1e3b1d24d89963 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 76c386bef61a4bd5a907266a5118ffe2; no voters:
I20240503 13:56:29.272626 16162 raft_consensus.cc:2781] T 97cd45c1b3b342a5aa1e3b1d24d89963 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:29.272686 16162 raft_consensus.cc:482] T 97cd45c1b3b342a5aa1e3b1d24d89963 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:29.272723 16162 raft_consensus.cc:3037] T 97cd45c1b3b342a5aa1e3b1d24d89963 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:29.272701 15889 raft_consensus.cc:2454] T 97cd45c1b3b342a5aa1e3b1d24d89963 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 0.
I20240503 13:56:29.274183 16162 raft_consensus.cc:504] T 97cd45c1b3b342a5aa1e3b1d24d89963 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:29.274626 16162 leader_election.cc:288] T 97cd45c1b3b342a5aa1e3b1d24d89963 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 election: Requested vote from peers 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:29.274968 15889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97cd45c1b3b342a5aa1e3b1d24d89963" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7"
I20240503 13:56:29.274968 15625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97cd45c1b3b342a5aa1e3b1d24d89963" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d"
I20240503 13:56:29.275091 15625 raft_consensus.cc:3037] T 97cd45c1b3b342a5aa1e3b1d24d89963 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:29.275096 15889 raft_consensus.cc:3037] T 97cd45c1b3b342a5aa1e3b1d24d89963 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:29.276090 15625 raft_consensus.cc:2454] T 97cd45c1b3b342a5aa1e3b1d24d89963 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 1.
I20240503 13:56:29.276105 15889 raft_consensus.cc:2454] T 97cd45c1b3b342a5aa1e3b1d24d89963 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 1.
I20240503 13:56:29.276372 15691 leader_election.cc:302] T 97cd45c1b3b342a5aa1e3b1d24d89963 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 76c386bef61a4bd5a907266a5118ffe2; no voters:
I20240503 13:56:29.276659 16162 raft_consensus.cc:2781] T 97cd45c1b3b342a5aa1e3b1d24d89963 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:29.276743 16162 raft_consensus.cc:686] T 97cd45c1b3b342a5aa1e3b1d24d89963 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 LEADER]: Becoming Leader. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Running, Role: LEADER
I20240503 13:56:29.276877 16162 consensus_queue.cc:237] T 97cd45c1b3b342a5aa1e3b1d24d89963 P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:29.278475 12911 catalog_manager.cc:5484] T 97cd45c1b3b342a5aa1e3b1d24d89963 P 76c386bef61a4bd5a907266a5118ffe2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71). New cstate: current_term: 1 leader_uuid: "76c386bef61a4bd5a907266a5118ffe2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:29.299660 15472 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:56:29.308719 13033 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:29.310245 13423 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 13:56:29.312882 15738 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:29.317587 15605 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:56:29.324265 15869 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:56:29.339498 13165 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:56:29.341082 13295 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240503 13:56:29.366888 15936 tablet.cc:2369] T 97cd45c1b3b342a5aa1e3b1d24d89963 P e47105a0f46e4511b7112e571688a4c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 13:56:29.397677 12911 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalance_test_table_2" } modify_external_catalogs: true
I20240503 13:56:29.397835 12911 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalance_test_table_2" } modify_external_catalogs: true
I20240503 13:56:29.400281 12911 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Sending DeleteTablet for 3 replicas of tablet 1be7152bed00461aa66db3ef430d7460
I20240503 13:56:29.400652 12911 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Sending DeleteTablet for 3 replicas of tablet b35569e6e1b34328bf334fe84551193b
I20240503 13:56:29.400893 12911 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Sending DeleteTablet for 3 replicas of tablet b9f328a343734ca89808aef24928c41b
I20240503 13:56:29.400974 13165 tablet_service.cc:1514] Processing DeleteTablet for tablet 1be7152bed00461aa66db3ef430d7460 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:29 UTC) from {username='slave'} at 127.0.0.1:41668
I20240503 13:56:29.401180 13423 tablet_service.cc:1514] Processing DeleteTablet for tablet 1be7152bed00461aa66db3ef430d7460 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:29 UTC) from {username='slave'} at 127.0.0.1:33678
I20240503 13:56:29.401307 13033 tablet_service.cc:1514] Processing DeleteTablet for tablet 1be7152bed00461aa66db3ef430d7460 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:29 UTC) from {username='slave'} at 127.0.0.1:59458
I20240503 13:56:29.401409 16092 tablet_replica.cc:330] T 1be7152bed00461aa66db3ef430d7460 P 8ccf5a4ac450414c9d3579bd5800b770: stopping tablet replica
I20240503 13:56:29.401441 16087 tablet_replica.cc:330] T 1be7152bed00461aa66db3ef430d7460 P 7690ee79577c45e3a2c8f8b552c0bff8: stopping tablet replica
I20240503 13:56:29.401494 13032 tablet_service.cc:1514] Processing DeleteTablet for tablet b35569e6e1b34328bf334fe84551193b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:29 UTC) from {username='slave'} at 127.0.0.1:59458
I20240503 13:56:29.401557 16092 raft_consensus.cc:2229] T 1be7152bed00461aa66db3ef430d7460 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:56:29.401614 16087 raft_consensus.cc:2229] T 1be7152bed00461aa66db3ef430d7460 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:29.401767 16087 raft_consensus.cc:2258] T 1be7152bed00461aa66db3ef430d7460 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:29.401836 16092 raft_consensus.cc:2258] T 1be7152bed00461aa66db3ef430d7460 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:29.402287 16092 ts_tablet_manager.cc:1875] T 1be7152bed00461aa66db3ef430d7460 P 8ccf5a4ac450414c9d3579bd5800b770: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:29.402356 16087 ts_tablet_manager.cc:1875] T 1be7152bed00461aa66db3ef430d7460 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:29.404072 16092 ts_tablet_manager.cc:1888] T 1be7152bed00461aa66db3ef430d7460 P 8ccf5a4ac450414c9d3579bd5800b770: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:56:29.404143 16092 log.cc:1196] T 1be7152bed00461aa66db3ef430d7460 P 8ccf5a4ac450414c9d3579bd5800b770: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/wal/wals/1be7152bed00461aa66db3ef430d7460
I20240503 13:56:29.404412 16092 ts_tablet_manager.cc:1909] T 1be7152bed00461aa66db3ef430d7460 P 8ccf5a4ac450414c9d3579bd5800b770: Deleting consensus metadata
I20240503 13:56:29.404532 16087 ts_tablet_manager.cc:1888] T 1be7152bed00461aa66db3ef430d7460 P 7690ee79577c45e3a2c8f8b552c0bff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:56:29.404639 16087 log.cc:1196] T 1be7152bed00461aa66db3ef430d7460 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal/wals/1be7152bed00461aa66db3ef430d7460
I20240503 13:56:29.404654 16085 tablet_replica.cc:330] T 1be7152bed00461aa66db3ef430d7460 P 60ba3ffe3b7746469834cd91c84be14e: stopping tablet replica
I20240503 13:56:29.404794 16085 raft_consensus.cc:2229] T 1be7152bed00461aa66db3ef430d7460 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:29.404891 16087 ts_tablet_manager.cc:1909] T 1be7152bed00461aa66db3ef430d7460 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting consensus metadata
I20240503 13:56:29.404937 16085 raft_consensus.cc:2258] T 1be7152bed00461aa66db3ef430d7460 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:29.405339 16085 ts_tablet_manager.cc:1875] T 1be7152bed00461aa66db3ef430d7460 P 60ba3ffe3b7746469834cd91c84be14e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:29.405817 12896 catalog_manager.cc:4830] TS 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109): tablet 1be7152bed00461aa66db3ef430d7460 (table rebalance_test_table_2 [id=16147ff5bb1c4ed9a5ab17905cb020ca]) successfully deleted
I20240503 13:56:29.406031 13427 tablet_service.cc:1514] Processing DeleteTablet for tablet b9f328a343734ca89808aef24928c41b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:29 UTC) from {username='slave'} at 127.0.0.1:33678
I20240503 13:56:29.406359 13032 tablet_service.cc:1514] Processing DeleteTablet for tablet b9f328a343734ca89808aef24928c41b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:29 UTC) from {username='slave'} at 127.0.0.1:59458
I20240503 13:56:29.406392 16087 tablet_replica.cc:330] T b9f328a343734ca89808aef24928c41b P 7690ee79577c45e3a2c8f8b552c0bff8: stopping tablet replica
I20240503 13:56:29.406534 16087 raft_consensus.cc:2229] T b9f328a343734ca89808aef24928c41b P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:56:29.406749 16087 raft_consensus.cc:2258] T b9f328a343734ca89808aef24928c41b P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:29.407263 16085 ts_tablet_manager.cc:1888] T 1be7152bed00461aa66db3ef430d7460 P 60ba3ffe3b7746469834cd91c84be14e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:56:29.407335 16085 log.cc:1196] T 1be7152bed00461aa66db3ef430d7460 P 60ba3ffe3b7746469834cd91c84be14e: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal/wals/1be7152bed00461aa66db3ef430d7460
I20240503 13:56:29.407310 16087 ts_tablet_manager.cc:1875] T b9f328a343734ca89808aef24928c41b P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:29.407545 16085 ts_tablet_manager.cc:1909] T 1be7152bed00461aa66db3ef430d7460 P 60ba3ffe3b7746469834cd91c84be14e: Deleting consensus metadata
I20240503 13:56:29.407970 13423 tablet_service.cc:1514] Processing DeleteTablet for tablet b35569e6e1b34328bf334fe84551193b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:29 UTC) from {username='slave'} at 127.0.0.1:33678
I20240503 13:56:29.408460 16085 tablet_replica.cc:330] T b35569e6e1b34328bf334fe84551193b P 60ba3ffe3b7746469834cd91c84be14e: stopping tablet replica
I20240503 13:56:29.408572 12897 catalog_manager.cc:4830] TS 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695): tablet 1be7152bed00461aa66db3ef430d7460 (table rebalance_test_table_2 [id=16147ff5bb1c4ed9a5ab17905cb020ca]) successfully deleted
I20240503 13:56:29.408671 12911 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:56:29.408757 16085 raft_consensus.cc:2229] T b35569e6e1b34328bf334fe84551193b P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:29.408902 16085 raft_consensus.cc:2258] T b35569e6e1b34328bf334fe84551193b P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:29.409235 16087 ts_tablet_manager.cc:1888] T b9f328a343734ca89808aef24928c41b P 7690ee79577c45e3a2c8f8b552c0bff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:56:29.409312 16087 log.cc:1196] T b9f328a343734ca89808aef24928c41b P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal/wals/b9f328a343734ca89808aef24928c41b
I20240503 13:56:29.409449 16085 ts_tablet_manager.cc:1875] T b35569e6e1b34328bf334fe84551193b P 60ba3ffe3b7746469834cd91c84be14e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:29.409591 16087 ts_tablet_manager.cc:1909] T b9f328a343734ca89808aef24928c41b P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting consensus metadata
I20240503 13:56:29.409896 13295 tablet_service.cc:1514] Processing DeleteTablet for tablet b35569e6e1b34328bf334fe84551193b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:29 UTC) from {username='slave'} at 127.0.0.1:33250
I20240503 13:56:29.410041 13165 tablet_service.cc:1514] Processing DeleteTablet for tablet b9f328a343734ca89808aef24928c41b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:29 UTC) from {username='slave'} at 127.0.0.1:41668
I20240503 13:56:29.410089 16086 tablet_replica.cc:330] T b35569e6e1b34328bf334fe84551193b P ba0194098aa84449ae17e8a0d4c43703: stopping tablet replica
I20240503 13:56:29.410229 16086 raft_consensus.cc:2229] T b35569e6e1b34328bf334fe84551193b P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:56:29.410549 16086 raft_consensus.cc:2258] T b35569e6e1b34328bf334fe84551193b P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:29.410769 12899 catalog_manager.cc:4830] TS 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387): tablet 1be7152bed00461aa66db3ef430d7460 (table rebalance_test_table_2 [id=16147ff5bb1c4ed9a5ab17905cb020ca]) successfully deleted
I20240503 13:56:29.410959 16092 tablet_replica.cc:330] T b9f328a343734ca89808aef24928c41b P 8ccf5a4ac450414c9d3579bd5800b770: stopping tablet replica
I20240503 13:56:29.411077 16092 raft_consensus.cc:2229] T b9f328a343734ca89808aef24928c41b P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:29.411077 16086 ts_tablet_manager.cc:1875] T b35569e6e1b34328bf334fe84551193b P ba0194098aa84449ae17e8a0d4c43703: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:29.411175 16092 raft_consensus.cc:2258] T b9f328a343734ca89808aef24928c41b P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:29.411408 16085 ts_tablet_manager.cc:1888] T b35569e6e1b34328bf334fe84551193b P 60ba3ffe3b7746469834cd91c84be14e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:56:29.411470 16085 log.cc:1196] T b35569e6e1b34328bf334fe84551193b P 60ba3ffe3b7746469834cd91c84be14e: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal/wals/b35569e6e1b34328bf334fe84551193b
I20240503 13:56:29.411904 16085 ts_tablet_manager.cc:1909] T b35569e6e1b34328bf334fe84551193b P 60ba3ffe3b7746469834cd91c84be14e: Deleting consensus metadata
I20240503 13:56:29.412541 16085 tablet_replica.cc:330] T b9f328a343734ca89808aef24928c41b P 60ba3ffe3b7746469834cd91c84be14e: stopping tablet replica
I20240503 13:56:29.412585 12897 catalog_manager.cc:4830] TS 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695): tablet b35569e6e1b34328bf334fe84551193b (table rebalance_test_table_2 [id=16147ff5bb1c4ed9a5ab17905cb020ca]) successfully deleted
I20240503 13:56:29.412667 16085 raft_consensus.cc:2229] T b9f328a343734ca89808aef24928c41b P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:29.412791 16085 raft_consensus.cc:2258] T b9f328a343734ca89808aef24928c41b P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:29.413275 16085 ts_tablet_manager.cc:1875] T b9f328a343734ca89808aef24928c41b P 60ba3ffe3b7746469834cd91c84be14e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:29.413873 15678 consensus_queue.cc:1035] T 356ccd6fdf25470dba2a79367a37128e P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:29.414567 16092 ts_tablet_manager.cc:1875] T b9f328a343734ca89808aef24928c41b P 8ccf5a4ac450414c9d3579bd5800b770: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:29.414911 15889 raft_consensus.cc:1261] T 97cd45c1b3b342a5aa1e3b1d24d89963 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Refusing update from remote peer 76c386bef61a4bd5a907266a5118ffe2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:29.414913 15625 raft_consensus.cc:1261] T 97cd45c1b3b342a5aa1e3b1d24d89963 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Refusing update from remote peer 76c386bef61a4bd5a907266a5118ffe2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:29.415535 15956 consensus_queue.cc:1035] T 97cd45c1b3b342a5aa1e3b1d24d89963 P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:29.415774 16086 ts_tablet_manager.cc:1888] T b35569e6e1b34328bf334fe84551193b P ba0194098aa84449ae17e8a0d4c43703: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:56:29.415774 16162 consensus_queue.cc:1035] T 97cd45c1b3b342a5aa1e3b1d24d89963 P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:29.415851 16086 log.cc:1196] T b35569e6e1b34328bf334fe84551193b P ba0194098aa84449ae17e8a0d4c43703: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/wal/wals/b35569e6e1b34328bf334fe84551193b
I20240503 13:56:29.416203 16086 ts_tablet_manager.cc:1909] T b35569e6e1b34328bf334fe84551193b P ba0194098aa84449ae17e8a0d4c43703: Deleting consensus metadata
I20240503 13:56:29.417579 16092 ts_tablet_manager.cc:1888] T b9f328a343734ca89808aef24928c41b P 8ccf5a4ac450414c9d3579bd5800b770: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:56:29.417680 16092 log.cc:1196] T b9f328a343734ca89808aef24928c41b P 8ccf5a4ac450414c9d3579bd5800b770: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/wal/wals/b9f328a343734ca89808aef24928c41b
I20240503 13:56:29.417797 12899 catalog_manager.cc:4830] TS ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279): tablet b35569e6e1b34328bf334fe84551193b (table rebalance_test_table_2 [id=16147ff5bb1c4ed9a5ab17905cb020ca]) successfully deleted
I20240503 13:56:29.417948 16092 ts_tablet_manager.cc:1909] T b9f328a343734ca89808aef24928c41b P 8ccf5a4ac450414c9d3579bd5800b770: Deleting consensus metadata
I20240503 13:56:29.418666 12896 catalog_manager.cc:4830] TS 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109): tablet b9f328a343734ca89808aef24928c41b (table rebalance_test_table_2 [id=16147ff5bb1c4ed9a5ab17905cb020ca]) successfully deleted
I20240503 13:56:29.418675 12899 catalog_manager.cc:4830] TS 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387): tablet b9f328a343734ca89808aef24928c41b (table rebalance_test_table_2 [id=16147ff5bb1c4ed9a5ab17905cb020ca]) successfully deleted
I20240503 13:56:29.419027 16087 tablet_replica.cc:330] T b35569e6e1b34328bf334fe84551193b P 7690ee79577c45e3a2c8f8b552c0bff8: stopping tablet replica
I20240503 13:56:29.419224 16087 raft_consensus.cc:2229] T b35569e6e1b34328bf334fe84551193b P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:29.419422 16087 raft_consensus.cc:2258] T b35569e6e1b34328bf334fe84551193b P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:29.420114 12911 catalog_manager.cc:5576] TS e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753) does not have the latest schema for tablet 97cd45c1b3b342a5aa1e3b1d24d89963 (table rebalancer_dynamic_table [id=4303f2a0523d40c090cfaaa50a92e519]). Expected version 1 got 0
I20240503 13:56:29.420212 16087 ts_tablet_manager.cc:1875] T b35569e6e1b34328bf334fe84551193b P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:29.420969 12909 catalog_manager.cc:5576] TS 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951) does not have the latest schema for tablet 97cd45c1b3b342a5aa1e3b1d24d89963 (table rebalancer_dynamic_table [id=4303f2a0523d40c090cfaaa50a92e519]). Expected version 1 got 0
I20240503 13:56:29.422518 16087 ts_tablet_manager.cc:1888] T b35569e6e1b34328bf334fe84551193b P 7690ee79577c45e3a2c8f8b552c0bff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:56:29.422602 16087 log.cc:1196] T b35569e6e1b34328bf334fe84551193b P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal/wals/b35569e6e1b34328bf334fe84551193b
I20240503 13:56:29.422909 16087 ts_tablet_manager.cc:1909] T b35569e6e1b34328bf334fe84551193b P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting consensus metadata
I20240503 13:56:29.423030 16186 tablet.cc:1722] T 97cd45c1b3b342a5aa1e3b1d24d89963 P 76c386bef61a4bd5a907266a5118ffe2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:29.423357 16186 tablet.cc:1620] T 97cd45c1b3b342a5aa1e3b1d24d89963 P 76c386bef61a4bd5a907266a5118ffe2: MemRowSet was empty: no flush needed.
I20240503 13:56:29.423555 12896 catalog_manager.cc:4830] TS 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109): tablet b35569e6e1b34328bf334fe84551193b (table rebalance_test_table_2 [id=16147ff5bb1c4ed9a5ab17905cb020ca]) successfully deleted
I20240503 13:56:29.425020 16187 tablet.cc:1722] T 97cd45c1b3b342a5aa1e3b1d24d89963 P 009f849ee7b54c80a3f5264695047e4d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:29.425992 12909 catalog_manager.cc:5576] TS 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951) does not have the latest schema for tablet 97cd45c1b3b342a5aa1e3b1d24d89963 (table rebalancer_dynamic_table [id=4303f2a0523d40c090cfaaa50a92e519]). Expected version 1 got 0
I20240503 13:56:29.426059 16187 tablet.cc:1620] T 97cd45c1b3b342a5aa1e3b1d24d89963 P 009f849ee7b54c80a3f5264695047e4d: MemRowSet was empty: no flush needed.
I20240503 13:56:29.427554 16183 mvcc.cc:205] Tried to move back new op lower bound from 7023593838235688960 to 7023593837679611904. Current Snapshot: MvccSnapshot[applied={T|T < 7023593838235688960}]
I20240503 13:56:29.427816 16085 ts_tablet_manager.cc:1888] T b9f328a343734ca89808aef24928c41b P 60ba3ffe3b7746469834cd91c84be14e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:56:29.428596 12911 catalog_manager.cc:5576] TS e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753) does not have the latest schema for tablet 97cd45c1b3b342a5aa1e3b1d24d89963 (table rebalancer_dynamic_table [id=4303f2a0523d40c090cfaaa50a92e519]). Expected version 1 got 0
I20240503 13:56:29.429608 16189 tablet.cc:1722] T 97cd45c1b3b342a5aa1e3b1d24d89963 P e47105a0f46e4511b7112e571688a4c7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:29.430363 16085 log.cc:1196] T b9f328a343734ca89808aef24928c41b P 60ba3ffe3b7746469834cd91c84be14e: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal/wals/b9f328a343734ca89808aef24928c41b
I20240503 13:56:29.430861 16085 ts_tablet_manager.cc:1909] T b9f328a343734ca89808aef24928c41b P 60ba3ffe3b7746469834cd91c84be14e: Deleting consensus metadata
I20240503 13:56:29.431352 16189 tablet.cc:1620] T 97cd45c1b3b342a5aa1e3b1d24d89963 P e47105a0f46e4511b7112e571688a4c7: MemRowSet was empty: no flush needed.
I20240503 13:56:29.431823 12897 catalog_manager.cc:4830] TS 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695): tablet b9f328a343734ca89808aef24928c41b (table rebalance_test_table_2 [id=16147ff5bb1c4ed9a5ab17905cb020ca]) successfully deleted
I20240503 13:56:29.431787 12911 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=4303f2a0523d40c090cfaaa50a92e519] alter complete (version 1)
I20240503 13:56:29.435392 15678 consensus_queue.cc:1035] T 356ccd6fdf25470dba2a79367a37128e P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:29.450735 13051 consensus_queue.cc:237] T b923fd3f226e409b813bb4e63325a7fc P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:29.452176 13315 raft_consensus.cc:1261] T b923fd3f226e409b813bb4e63325a7fc P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Refusing update from remote peer 60ba3ffe3b7746469834cd91c84be14e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:56:29.452628 13446 raft_consensus.cc:1261] T b923fd3f226e409b813bb4e63325a7fc P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer 60ba3ffe3b7746469834cd91c84be14e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:56:29.452809 16205 consensus_queue.cc:1035] T b923fd3f226e409b813bb4e63325a7fc P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:29.452998 15332 consensus_queue.cc:1035] T b923fd3f226e409b813bb4e63325a7fc P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:29.455936 15332 raft_consensus.cc:2932] T b923fd3f226e409b813bb4e63325a7fc P 60ba3ffe3b7746469834cd91c84be14e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
W20240503 13:56:29.456743 12987 consensus_peers.cc:485] T b923fd3f226e409b813bb4e63325a7fc P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b923fd3f226e409b813bb4e63325a7fc. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:29.456250 13446 raft_consensus.cc:2932] T b923fd3f226e409b813bb4e63325a7fc P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:29.458315 13315 raft_consensus.cc:2932] T b923fd3f226e409b813bb4e63325a7fc P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:29.459834 12911 catalog_manager.cc:5484] T b923fd3f226e409b813bb4e63325a7fc P 60ba3ffe3b7746469834cd91c84be14e reported cstate change: config changed from index -1 to 4, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) added. New cstate: current_term: 1 leader_uuid: "60ba3ffe3b7746469834cd91c84be14e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:29.480239 13315 consensus_queue.cc:237] T fd6a3d908840456f82c346c4bf1d8160 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:29.481942 13446 raft_consensus.cc:1261] T fd6a3d908840456f82c346c4bf1d8160 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:56:29.482394 16225 consensus_queue.cc:1035] T fd6a3d908840456f82c346c4bf1d8160 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:29.482609 13051 raft_consensus.cc:1261] T fd6a3d908840456f82c346c4bf1d8160 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:56:29.483064 16225 consensus_queue.cc:1035] T fd6a3d908840456f82c346c4bf1d8160 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:29.485265 15154 raft_consensus.cc:2932] T fd6a3d908840456f82c346c4bf1d8160 P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 61b887f0d40d497badeca5604c1f0546 (127.6.98.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
W20240503 13:56:29.485546 13252 consensus_peers.cc:485] T fd6a3d908840456f82c346c4bf1d8160 P ba0194098aa84449ae17e8a0d4c43703 -> Peer 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153): Couldn't send request to peer 61b887f0d40d497badeca5604c1f0546. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fd6a3d908840456f82c346c4bf1d8160. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:29.485769 13051 raft_consensus.cc:2932] T fd6a3d908840456f82c346c4bf1d8160 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 61b887f0d40d497badeca5604c1f0546 (127.6.98.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:29.487689 13447 raft_consensus.cc:2932] T fd6a3d908840456f82c346c4bf1d8160 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 61b887f0d40d497badeca5604c1f0546 (127.6.98.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:29.488463 12908 catalog_manager.cc:5484] T fd6a3d908840456f82c346c4bf1d8160 P 60ba3ffe3b7746469834cd91c84be14e reported cstate change: config changed from index -1 to 4, VOTER 61b887f0d40d497badeca5604c1f0546 (127.6.98.69) added. New cstate: current_term: 1 leader_uuid: "ba0194098aa84449ae17e8a0d4c43703" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:29.505908 16119 consensus_queue.cc:1035] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:29.509719 13315 consensus_queue.cc:237] T 4647faf6a457415d8ae2b6691f452eab P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:29.510346 16103 consensus_queue.cc:1035] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:29.511611 13184 raft_consensus.cc:1261] T 4647faf6a457415d8ae2b6691f452eab P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:56:29.512053 13447 raft_consensus.cc:1261] T 4647faf6a457415d8ae2b6691f452eab P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:56:29.512392 15152 consensus_queue.cc:1035] T 4647faf6a457415d8ae2b6691f452eab P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:29.512684 16225 consensus_queue.cc:1035] T 4647faf6a457415d8ae2b6691f452eab P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:29.515167 15152 raft_consensus.cc:2932] T 4647faf6a457415d8ae2b6691f452eab P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 009f849ee7b54c80a3f5264695047e4d (127.6.98.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:29.515600 13184 raft_consensus.cc:2932] T 4647faf6a457415d8ae2b6691f452eab P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 009f849ee7b54c80a3f5264695047e4d (127.6.98.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
W20240503 13:56:29.516263 13249 consensus_peers.cc:485] T 4647faf6a457415d8ae2b6691f452eab P ba0194098aa84449ae17e8a0d4c43703 -> Peer 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951): Couldn't send request to peer 009f849ee7b54c80a3f5264695047e4d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4647faf6a457415d8ae2b6691f452eab. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:29.517244 13447 raft_consensus.cc:2932] T 4647faf6a457415d8ae2b6691f452eab P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 009f849ee7b54c80a3f5264695047e4d (127.6.98.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:29.518497 12908 catalog_manager.cc:5484] T 4647faf6a457415d8ae2b6691f452eab P ba0194098aa84449ae17e8a0d4c43703 reported cstate change: config changed from index -1 to 4, VOTER 009f849ee7b54c80a3f5264695047e4d (127.6.98.70) added. New cstate: current_term: 1 leader_uuid: "ba0194098aa84449ae17e8a0d4c43703" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:29.535969 12908 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:56:29.542237 16186 tablet.cc:1722] T 97cd45c1b3b342a5aa1e3b1d24d89963 P 76c386bef61a4bd5a907266a5118ffe2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:29.542523 16189 tablet.cc:1722] T 97cd45c1b3b342a5aa1e3b1d24d89963 P e47105a0f46e4511b7112e571688a4c7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:29.542609 16186 tablet.cc:1620] T 97cd45c1b3b342a5aa1e3b1d24d89963 P 76c386bef61a4bd5a907266a5118ffe2: MemRowSet was empty: no flush needed.
I20240503 13:56:29.542757 16187 tablet.cc:1722] T 97cd45c1b3b342a5aa1e3b1d24d89963 P 009f849ee7b54c80a3f5264695047e4d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:29.542819 16189 tablet.cc:1620] T 97cd45c1b3b342a5aa1e3b1d24d89963 P e47105a0f46e4511b7112e571688a4c7: MemRowSet was empty: no flush needed.
I20240503 13:56:29.543037 16187 tablet.cc:1620] T 97cd45c1b3b342a5aa1e3b1d24d89963 P 009f849ee7b54c80a3f5264695047e4d: MemRowSet was empty: no flush needed.
I20240503 13:56:29.548018 13447 consensus_queue.cc:237] T 81d47f705c0b4128a84e4bf7d274a659 P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:29.548377 12911 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=4303f2a0523d40c090cfaaa50a92e519] alter complete (version 2)
I20240503 13:56:29.549880 13051 raft_consensus.cc:1261] T 81d47f705c0b4128a84e4bf7d274a659 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Refusing update from remote peer 7690ee79577c45e3a2c8f8b552c0bff8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:56:29.550259 13315 raft_consensus.cc:1261] T 81d47f705c0b4128a84e4bf7d274a659 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Refusing update from remote peer 7690ee79577c45e3a2c8f8b552c0bff8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:56:29.550727 15295 consensus_queue.cc:1035] T 81d47f705c0b4128a84e4bf7d274a659 P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 13:56:29.550954 13383 consensus_peers.cc:485] T 81d47f705c0b4128a84e4bf7d274a659 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885): Couldn't send request to peer 76c386bef61a4bd5a907266a5118ffe2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 81d47f705c0b4128a84e4bf7d274a659. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:29.551163 16268 consensus_queue.cc:1035] T 81d47f705c0b4128a84e4bf7d274a659 P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:29.553002 15295 raft_consensus.cc:2932] T 81d47f705c0b4128a84e4bf7d274a659 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:29.554872 13315 raft_consensus.cc:2932] T 81d47f705c0b4128a84e4bf7d274a659 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:29.555908 12905 catalog_manager.cc:5484] T 81d47f705c0b4128a84e4bf7d274a659 P 7690ee79577c45e3a2c8f8b552c0bff8 reported cstate change: config changed from index -1 to 4, VOTER 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71) added. New cstate: current_term: 1 leader_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:29.556083 13052 raft_consensus.cc:2932] T 81d47f705c0b4128a84e4bf7d274a659 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:29.582530 13052 consensus_queue.cc:237] T 4bd42569fd3b4b39b0bd489acebbad10 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:29.583866 13447 raft_consensus.cc:1261] T 4bd42569fd3b4b39b0bd489acebbad10 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer 60ba3ffe3b7746469834cd91c84be14e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:56:29.583866 13184 raft_consensus.cc:1261] T 4bd42569fd3b4b39b0bd489acebbad10 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Refusing update from remote peer 60ba3ffe3b7746469834cd91c84be14e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 13:56:29.584196 12987 consensus_peers.cc:485] T 4bd42569fd3b4b39b0bd489acebbad10 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4bd42569fd3b4b39b0bd489acebbad10. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:29.584342 16205 consensus_queue.cc:1035] T 4bd42569fd3b4b39b0bd489acebbad10 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:29.584556 15332 consensus_queue.cc:1035] T 4bd42569fd3b4b39b0bd489acebbad10 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:29.586356 16205 raft_consensus.cc:2932] T 4bd42569fd3b4b39b0bd489acebbad10 P 60ba3ffe3b7746469834cd91c84be14e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:29.586788 13184 raft_consensus.cc:2932] T 4bd42569fd3b4b39b0bd489acebbad10 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:29.588317 13447 raft_consensus.cc:2932] T 4bd42569fd3b4b39b0bd489acebbad10 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:29.589633 12908 catalog_manager.cc:5484] T 4bd42569fd3b4b39b0bd489acebbad10 P 8ccf5a4ac450414c9d3579bd5800b770 reported cstate change: config changed from index -1 to 4, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) added. New cstate: current_term: 1 leader_uuid: "60ba3ffe3b7746469834cd91c84be14e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:29.609690 13447 consensus_queue.cc:237] T b291e3dd6a7a454ca4d736eff8dd3a41 P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:29.611143 13315 raft_consensus.cc:1261] T b291e3dd6a7a454ca4d736eff8dd3a41 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Refusing update from remote peer 7690ee79577c45e3a2c8f8b552c0bff8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:56:29.611171 13184 raft_consensus.cc:1261] T b291e3dd6a7a454ca4d736eff8dd3a41 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Refusing update from remote peer 7690ee79577c45e3a2c8f8b552c0bff8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 13:56:29.611256 13383 consensus_peers.cc:485] T b291e3dd6a7a454ca4d736eff8dd3a41 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153): Couldn't send request to peer 61b887f0d40d497badeca5604c1f0546. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b291e3dd6a7a454ca4d736eff8dd3a41. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:29.611533 15678 consensus_queue.cc:1035] T b291e3dd6a7a454ca4d736eff8dd3a41 P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:29.611709 16268 consensus_queue.cc:1035] T b291e3dd6a7a454ca4d736eff8dd3a41 P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:29.613633 15678 raft_consensus.cc:2932] T b291e3dd6a7a454ca4d736eff8dd3a41 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 61b887f0d40d497badeca5604c1f0546 (127.6.98.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:29.614037 13184 raft_consensus.cc:2932] T b291e3dd6a7a454ca4d736eff8dd3a41 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 61b887f0d40d497badeca5604c1f0546 (127.6.98.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:29.616839 12905 catalog_manager.cc:5484] T b291e3dd6a7a454ca4d736eff8dd3a41 P 8ccf5a4ac450414c9d3579bd5800b770 reported cstate change: config changed from index -1 to 4, VOTER 61b887f0d40d497badeca5604c1f0546 (127.6.98.69) added. New cstate: current_term: 1 leader_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:29.619783 13315 raft_consensus.cc:2932] T b291e3dd6a7a454ca4d736eff8dd3a41 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 61b887f0d40d497badeca5604c1f0546 (127.6.98.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:29.650868 13315 consensus_queue.cc:237] T b47b28a4766a425e8b9bdfea882f979d P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:29.651084 12905 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:56:29.652370 13052 raft_consensus.cc:1261] T b47b28a4766a425e8b9bdfea882f979d P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:56:29.652864 13184 raft_consensus.cc:1261] T b47b28a4766a425e8b9bdfea882f979d P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:56:29.653296 16229 consensus_queue.cc:1035] T b47b28a4766a425e8b9bdfea882f979d P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:29.655287 16229 consensus_queue.cc:1035] T b47b28a4766a425e8b9bdfea882f979d P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 13:56:29.656806 13252 consensus_peers.cc:485] T b47b28a4766a425e8b9bdfea882f979d P ba0194098aa84449ae17e8a0d4c43703 -> Peer 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885): Couldn't send request to peer 76c386bef61a4bd5a907266a5118ffe2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b47b28a4766a425e8b9bdfea882f979d. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:29.657850 16186 tablet.cc:1722] T 97cd45c1b3b342a5aa1e3b1d24d89963 P 76c386bef61a4bd5a907266a5118ffe2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:29.658376 16189 tablet.cc:1722] T 97cd45c1b3b342a5aa1e3b1d24d89963 P e47105a0f46e4511b7112e571688a4c7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:29.658720 16187 tablet.cc:1722] T 97cd45c1b3b342a5aa1e3b1d24d89963 P 009f849ee7b54c80a3f5264695047e4d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:29.662631 16229 raft_consensus.cc:2932] T b47b28a4766a425e8b9bdfea882f979d P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:29.662999 12911 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table_new_name [id=4303f2a0523d40c090cfaaa50a92e519] alter complete (version 3)
I20240503 13:56:29.663141 13052 raft_consensus.cc:2932] T b47b28a4766a425e8b9bdfea882f979d P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:29.665444 12909 catalog_manager.cc:5484] T b47b28a4766a425e8b9bdfea882f979d P ba0194098aa84449ae17e8a0d4c43703 reported cstate change: config changed from index -1 to 4, VOTER 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71) added. New cstate: current_term: 1 leader_uuid: "ba0194098aa84449ae17e8a0d4c43703" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:29.665729 13184 raft_consensus.cc:2932] T b47b28a4766a425e8b9bdfea882f979d P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:29.666605 16162 consensus_queue.cc:1035] T 0878ccb6135f4485b5ba35ae67a76d0c P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:29.669952 15956 consensus_queue.cc:1035] T 0878ccb6135f4485b5ba35ae67a76d0c P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:29.689513 13184 consensus_queue.cc:237] T b26d8338f20749ed92f5d7670e9549d5 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:29.691089 13052 raft_consensus.cc:1261] T b26d8338f20749ed92f5d7670e9549d5 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Refusing update from remote peer 8ccf5a4ac450414c9d3579bd5800b770: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:56:29.691604 16091 consensus_queue.cc:1035] T b26d8338f20749ed92f5d7670e9549d5 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:29.691761 13447 raft_consensus.cc:1261] T b26d8338f20749ed92f5d7670e9549d5 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer 8ccf5a4ac450414c9d3579bd5800b770: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:56:29.692622 16091 consensus_queue.cc:1035] T b26d8338f20749ed92f5d7670e9549d5 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:29.694547 16198 raft_consensus.cc:2932] T b26d8338f20749ed92f5d7670e9549d5 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 009f849ee7b54c80a3f5264695047e4d (127.6.98.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:29.694955 13447 raft_consensus.cc:2932] T b26d8338f20749ed92f5d7670e9549d5 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 009f849ee7b54c80a3f5264695047e4d (127.6.98.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
W20240503 13:56:29.695380 13118 consensus_peers.cc:485] T b26d8338f20749ed92f5d7670e9549d5 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951): Couldn't send request to peer 009f849ee7b54c80a3f5264695047e4d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b26d8338f20749ed92f5d7670e9549d5. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:29.696561 13052 raft_consensus.cc:2932] T b26d8338f20749ed92f5d7670e9549d5 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 009f849ee7b54c80a3f5264695047e4d (127.6.98.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:29.699080 12905 catalog_manager.cc:5484] T b26d8338f20749ed92f5d7670e9549d5 P 7690ee79577c45e3a2c8f8b552c0bff8 reported cstate change: config changed from index -1 to 4, VOTER 009f849ee7b54c80a3f5264695047e4d (127.6.98.70) added. New cstate: current_term: 1 leader_uuid: "8ccf5a4ac450414c9d3579bd5800b770" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:29.717098 13315 consensus_queue.cc:237] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:29.718554 13184 raft_consensus.cc:1261] T a746bcb2d718433f8c9acc8bd027a429 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:56:29.718538 13447 raft_consensus.cc:1261] T a746bcb2d718433f8c9acc8bd027a429 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:56:29.718953 15154 consensus_queue.cc:1035] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:29.719117 16225 consensus_queue.cc:1035] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:29.721388 16225 raft_consensus.cc:2932] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:29.724301 13184 raft_consensus.cc:2932] T a746bcb2d718433f8c9acc8bd027a429 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:29.724444 13447 raft_consensus.cc:2932] T a746bcb2d718433f8c9acc8bd027a429 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:29.725342 12908 catalog_manager.cc:5484] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 reported cstate change: config changed from index -1 to 4, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) added. New cstate: current_term: 1 leader_uuid: "ba0194098aa84449ae17e8a0d4c43703" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:29.729442 12911 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_extra_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 13:56:29.730046 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a746bcb2d718433f8c9acc8bd027a429. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:29.738025 15869 tablet_service.cc:1467] Processing CreateTablet for tablet a3c4cbf779554d8b84a48de78a9c392e (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=65fb8aeaab8a4eb884e47e4e0f7ef87d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:29.738445 15869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3c4cbf779554d8b84a48de78a9c392e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:29.738481 13165 tablet_service.cc:1467] Processing CreateTablet for tablet a3c4cbf779554d8b84a48de78a9c392e (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=65fb8aeaab8a4eb884e47e4e0f7ef87d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:29.738521 15472 tablet_service.cc:1467] Processing CreateTablet for tablet a3c4cbf779554d8b84a48de78a9c392e (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=65fb8aeaab8a4eb884e47e4e0f7ef87d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:29.738718 13165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3c4cbf779554d8b84a48de78a9c392e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:29.738827 15472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3c4cbf779554d8b84a48de78a9c392e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:29.741115 15737 tablet_service.cc:1467] Processing CreateTablet for tablet 0f9224c2e3b946d993a5fa5c95a1d681 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=65fb8aeaab8a4eb884e47e4e0f7ef87d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:29.741407 15737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f9224c2e3b946d993a5fa5c95a1d681. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:29.741780 15605 tablet_service.cc:1467] Processing CreateTablet for tablet 7bdb9c14caf04abca39e56f8bcf1d1d4 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=65fb8aeaab8a4eb884e47e4e0f7ef87d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:29.742039 15605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bdb9c14caf04abca39e56f8bcf1d1d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:29.742158 16379 tablet_bootstrap.cc:492] T a3c4cbf779554d8b84a48de78a9c392e P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:29.742851 15869 tablet_service.cc:1467] Processing CreateTablet for tablet 7bdb9c14caf04abca39e56f8bcf1d1d4 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=65fb8aeaab8a4eb884e47e4e0f7ef87d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:29.743105 15869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bdb9c14caf04abca39e56f8bcf1d1d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:29.743398 16379 tablet_bootstrap.cc:654] T a3c4cbf779554d8b84a48de78a9c392e P e47105a0f46e4511b7112e571688a4c7: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:29.741124 15738 tablet_service.cc:1467] Processing CreateTablet for tablet 7bdb9c14caf04abca39e56f8bcf1d1d4 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=65fb8aeaab8a4eb884e47e4e0f7ef87d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:29.744392 15738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bdb9c14caf04abca39e56f8bcf1d1d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:29.744442 16373 tablet_bootstrap.cc:492] T a3c4cbf779554d8b84a48de78a9c392e P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:29.744607 15604 tablet_service.cc:1467] Processing CreateTablet for tablet 0f9224c2e3b946d993a5fa5c95a1d681 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=65fb8aeaab8a4eb884e47e4e0f7ef87d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:29.745008 15604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f9224c2e3b946d993a5fa5c95a1d681. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:29.745859 16373 tablet_bootstrap.cc:654] T a3c4cbf779554d8b84a48de78a9c392e P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:29.745857 15868 tablet_service.cc:1467] Processing CreateTablet for tablet 0f9224c2e3b946d993a5fa5c95a1d681 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=65fb8aeaab8a4eb884e47e4e0f7ef87d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:29.746109 15868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f9224c2e3b946d993a5fa5c95a1d681. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:29.746521 16380 tablet_bootstrap.cc:492] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:29.747821 16380 tablet_bootstrap.cc:654] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:29.749059 16379 tablet_bootstrap.cc:492] T a3c4cbf779554d8b84a48de78a9c392e P e47105a0f46e4511b7112e571688a4c7: No bootstrap required, opened a new log
I20240503 13:56:29.749182 16379 ts_tablet_manager.cc:1367] T a3c4cbf779554d8b84a48de78a9c392e P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 13:56:29.749521 16380 tablet_bootstrap.cc:492] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:29.749905 16380 ts_tablet_manager.cc:1367] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:56:29.750249 16374 tablet_bootstrap.cc:492] T a3c4cbf779554d8b84a48de78a9c392e P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:29.750466 16379 raft_consensus.cc:348] T a3c4cbf779554d8b84a48de78a9c392e P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:29.750469 16380 raft_consensus.cc:348] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:29.750578 16380 raft_consensus.cc:374] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:29.750615 16379 raft_consensus.cc:374] T a3c4cbf779554d8b84a48de78a9c392e P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:29.750639 16380 raft_consensus.cc:729] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:29.750667 16379 raft_consensus.cc:729] T a3c4cbf779554d8b84a48de78a9c392e P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:29.750805 16380 consensus_queue.cc:260] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:29.750825 16379 consensus_queue.cc:260] T a3c4cbf779554d8b84a48de78a9c392e P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:29.751104 16380 ts_tablet_manager.cc:1398] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:29.751252 16380 tablet_bootstrap.cc:492] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:29.751544 16374 tablet_bootstrap.cc:654] T a3c4cbf779554d8b84a48de78a9c392e P 61b887f0d40d497badeca5604c1f0546: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:29.752139 16380 tablet_bootstrap.cc:654] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:29.753418 16380 tablet_bootstrap.cc:492] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:29.753508 16380 ts_tablet_manager.cc:1367] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:56:29.753945 16380 raft_consensus.cc:348] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:29.754079 16380 raft_consensus.cc:374] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:29.754137 16380 raft_consensus.cc:729] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:29.754258 16380 consensus_queue.cc:260] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:29.754571 16380 ts_tablet_manager.cc:1398] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:29.756825 16379 ts_tablet_manager.cc:1398] T a3c4cbf779554d8b84a48de78a9c392e P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240503 13:56:29.757789 16379 tablet_bootstrap.cc:492] T 0f9224c2e3b946d993a5fa5c95a1d681 P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:29.759344 16379 tablet_bootstrap.cc:654] T 0f9224c2e3b946d993a5fa5c95a1d681 P e47105a0f46e4511b7112e571688a4c7: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:29.760155 16381 tablet_bootstrap.cc:492] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:29.761034 16379 tablet_bootstrap.cc:492] T 0f9224c2e3b946d993a5fa5c95a1d681 P e47105a0f46e4511b7112e571688a4c7: No bootstrap required, opened a new log
I20240503 13:56:29.761185 16379 ts_tablet_manager.cc:1367] T 0f9224c2e3b946d993a5fa5c95a1d681 P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:56:29.761694 16379 raft_consensus.cc:348] T 0f9224c2e3b946d993a5fa5c95a1d681 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:29.761794 16379 raft_consensus.cc:374] T 0f9224c2e3b946d993a5fa5c95a1d681 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:29.761822 16379 raft_consensus.cc:729] T 0f9224c2e3b946d993a5fa5c95a1d681 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:29.762133 16379 consensus_queue.cc:260] T 0f9224c2e3b946d993a5fa5c95a1d681 P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:29.762567 16379 ts_tablet_manager.cc:1398] T 0f9224c2e3b946d993a5fa5c95a1d681 P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:29.762773 16379 tablet_bootstrap.cc:492] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:29.763203 16381 tablet_bootstrap.cc:654] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:29.763703 16379 tablet_bootstrap.cc:654] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P e47105a0f46e4511b7112e571688a4c7: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:29.764168 16373 tablet_bootstrap.cc:492] T a3c4cbf779554d8b84a48de78a9c392e P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:29.764304 16373 ts_tablet_manager.cc:1367] T a3c4cbf779554d8b84a48de78a9c392e P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.002s
I20240503 13:56:29.764832 16379 tablet_bootstrap.cc:492] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P e47105a0f46e4511b7112e571688a4c7: No bootstrap required, opened a new log
I20240503 13:56:29.764907 16379 ts_tablet_manager.cc:1367] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:29.765316 16379 raft_consensus.cc:348] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:29.765411 16379 raft_consensus.cc:374] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:29.765439 16379 raft_consensus.cc:729] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:29.765547 16379 consensus_queue.cc:260] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:29.765625 16373 raft_consensus.cc:348] T a3c4cbf779554d8b84a48de78a9c392e P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:29.765730 16373 raft_consensus.cc:374] T a3c4cbf779554d8b84a48de78a9c392e P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:29.765758 16373 raft_consensus.cc:729] T a3c4cbf779554d8b84a48de78a9c392e P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:29.765794 16379 ts_tablet_manager.cc:1398] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:29.765897 16373 consensus_queue.cc:260] T a3c4cbf779554d8b84a48de78a9c392e P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:29.766340 16373 ts_tablet_manager.cc:1398] T a3c4cbf779554d8b84a48de78a9c392e P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:29.766726 12911 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:29.766853 12911 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:29.768889 16374 tablet_bootstrap.cc:492] T a3c4cbf779554d8b84a48de78a9c392e P 61b887f0d40d497badeca5604c1f0546: No bootstrap required, opened a new log
I20240503 13:56:29.768996 16374 ts_tablet_manager.cc:1367] T a3c4cbf779554d8b84a48de78a9c392e P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240503 13:56:29.769208 12911 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Sending DeleteTablet for 3 replicas of tablet 97cd45c1b3b342a5aa1e3b1d24d89963
I20240503 13:56:29.769512 16374 raft_consensus.cc:348] T a3c4cbf779554d8b84a48de78a9c392e P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:29.769727 16374 raft_consensus.cc:374] T a3c4cbf779554d8b84a48de78a9c392e P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:29.769764 16374 raft_consensus.cc:729] T a3c4cbf779554d8b84a48de78a9c392e P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:29.769896 16374 consensus_queue.cc:260] T a3c4cbf779554d8b84a48de78a9c392e P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:29.770117 15869 tablet_service.cc:1514] Processing DeleteTablet for tablet 97cd45c1b3b342a5aa1e3b1d24d89963 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:29 UTC) from {username='slave'} at 127.0.0.1:50498
I20240503 13:56:29.770236 15738 tablet_service.cc:1514] Processing DeleteTablet for tablet 97cd45c1b3b342a5aa1e3b1d24d89963 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:29 UTC) from {username='slave'} at 127.0.0.1:44984
I20240503 13:56:29.770282 16381 tablet_bootstrap.cc:492] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:29.770292 16374 ts_tablet_manager.cc:1398] T a3c4cbf779554d8b84a48de78a9c392e P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:29.770368 16381 ts_tablet_manager.cc:1367] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 13:56:29.770733 15604 tablet_service.cc:1514] Processing DeleteTablet for tablet 97cd45c1b3b342a5aa1e3b1d24d89963 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:29 UTC) from {username='slave'} at 127.0.0.1:60718
I20240503 13:56:29.770840 16381 raft_consensus.cc:348] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:29.770965 16381 raft_consensus.cc:374] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:29.770994 16381 raft_consensus.cc:729] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:29.771117 16381 consensus_queue.cc:260] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:29.771390 16381 ts_tablet_manager.cc:1398] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:29.771584 16381 tablet_bootstrap.cc:492] T 0f9224c2e3b946d993a5fa5c95a1d681 P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:29.771580 16397 tablet_replica.cc:330] T 97cd45c1b3b342a5aa1e3b1d24d89963 P e47105a0f46e4511b7112e571688a4c7: stopping tablet replica
I20240503 13:56:29.772053 16397 raft_consensus.cc:2229] T 97cd45c1b3b342a5aa1e3b1d24d89963 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:29.772262 16397 raft_consensus.cc:2258] T 97cd45c1b3b342a5aa1e3b1d24d89963 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:29.772607 16398 tablet_replica.cc:330] T 97cd45c1b3b342a5aa1e3b1d24d89963 P 009f849ee7b54c80a3f5264695047e4d: stopping tablet replica
I20240503 13:56:29.772843 16398 raft_consensus.cc:2229] T 97cd45c1b3b342a5aa1e3b1d24d89963 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:29.773048 16398 raft_consensus.cc:2258] T 97cd45c1b3b342a5aa1e3b1d24d89963 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:29.773566 16381 tablet_bootstrap.cc:654] T 0f9224c2e3b946d993a5fa5c95a1d681 P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:29.773929 16397 ts_tablet_manager.cc:1875] T 97cd45c1b3b342a5aa1e3b1d24d89963 P e47105a0f46e4511b7112e571688a4c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:29.774909 16381 tablet_bootstrap.cc:492] T 0f9224c2e3b946d993a5fa5c95a1d681 P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:29.774986 16381 ts_tablet_manager.cc:1367] T 0f9224c2e3b946d993a5fa5c95a1d681 P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 13:56:29.775138 16398 ts_tablet_manager.cc:1875] T 97cd45c1b3b342a5aa1e3b1d24d89963 P 009f849ee7b54c80a3f5264695047e4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:29.775429 16381 raft_consensus.cc:348] T 0f9224c2e3b946d993a5fa5c95a1d681 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:29.775525 16381 raft_consensus.cc:374] T 0f9224c2e3b946d993a5fa5c95a1d681 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:29.775563 16381 raft_consensus.cc:729] T 0f9224c2e3b946d993a5fa5c95a1d681 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:29.775674 16381 consensus_queue.cc:260] T 0f9224c2e3b946d993a5fa5c95a1d681 P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:29.775925 16381 ts_tablet_manager.cc:1398] T 0f9224c2e3b946d993a5fa5c95a1d681 P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:29.776844 16397 ts_tablet_manager.cc:1888] T 97cd45c1b3b342a5aa1e3b1d24d89963 P e47105a0f46e4511b7112e571688a4c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:29.776993 16396 tablet_replica.cc:330] T 97cd45c1b3b342a5aa1e3b1d24d89963 P 76c386bef61a4bd5a907266a5118ffe2: stopping tablet replica
I20240503 13:56:29.777009 16397 log.cc:1196] T 97cd45c1b3b342a5aa1e3b1d24d89963 P e47105a0f46e4511b7112e571688a4c7: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-7/wal/wals/97cd45c1b3b342a5aa1e3b1d24d89963
I20240503 13:56:29.777169 16396 raft_consensus.cc:2229] T 97cd45c1b3b342a5aa1e3b1d24d89963 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:56:29.777346 16397 ts_tablet_manager.cc:1909] T 97cd45c1b3b342a5aa1e3b1d24d89963 P e47105a0f46e4511b7112e571688a4c7: Deleting consensus metadata
I20240503 13:56:29.778007 16398 ts_tablet_manager.cc:1888] T 97cd45c1b3b342a5aa1e3b1d24d89963 P 009f849ee7b54c80a3f5264695047e4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:29.778092 16398 log.cc:1196] T 97cd45c1b3b342a5aa1e3b1d24d89963 P 009f849ee7b54c80a3f5264695047e4d: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/wal/wals/97cd45c1b3b342a5aa1e3b1d24d89963
I20240503 13:56:29.778364 16161 raft_consensus.cc:482] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:29.778457 16161 raft_consensus.cc:504] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:29.778539 12896 catalog_manager.cc:4830] TS e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): tablet 97cd45c1b3b342a5aa1e3b1d24d89963 (table rebalancer_dynamic_table_new_name [id=4303f2a0523d40c090cfaaa50a92e519]) successfully deleted
I20240503 13:56:29.779438 16161 leader_election.cc:288] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951), 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885)
I20240503 13:56:29.779706 16398 ts_tablet_manager.cc:1909] T 97cd45c1b3b342a5aa1e3b1d24d89963 P 009f849ee7b54c80a3f5264695047e4d: Deleting consensus metadata
I20240503 13:56:29.780036 16396 raft_consensus.cc:2258] T 97cd45c1b3b342a5aa1e3b1d24d89963 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:29.780723 12896 catalog_manager.cc:4830] TS 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951): tablet 97cd45c1b3b342a5aa1e3b1d24d89963 (table rebalancer_dynamic_table_new_name [id=4303f2a0523d40c090cfaaa50a92e519]) successfully deleted
I20240503 13:56:29.780900 16396 ts_tablet_manager.cc:1875] T 97cd45c1b3b342a5aa1e3b1d24d89963 P 76c386bef61a4bd5a907266a5118ffe2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:29.784657 15625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bdb9c14caf04abca39e56f8bcf1d1d4" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d" is_pre_election: true
I20240503 13:56:29.784798 15625 raft_consensus.cc:2454] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 0.
I20240503 13:56:29.784816 13315 consensus_queue.cc:237] T 9eb0d911701b4b2bb33a6b7ab59f08aa P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:29.785064 16396 ts_tablet_manager.cc:1888] T 97cd45c1b3b342a5aa1e3b1d24d89963 P 76c386bef61a4bd5a907266a5118ffe2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:29.785144 16396 log.cc:1196] T 97cd45c1b3b342a5aa1e3b1d24d89963 P 76c386bef61a4bd5a907266a5118ffe2: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/wal/wals/97cd45c1b3b342a5aa1e3b1d24d89963
I20240503 13:56:29.785122 15822 leader_election.cc:302] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:29.785410 16396 ts_tablet_manager.cc:1909] T 97cd45c1b3b342a5aa1e3b1d24d89963 P 76c386bef61a4bd5a907266a5118ffe2: Deleting consensus metadata
I20240503 13:56:29.785408 16161 raft_consensus.cc:2781] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:29.785471 16161 raft_consensus.cc:482] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:29.785507 16161 raft_consensus.cc:3037] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:29.786445 16161 raft_consensus.cc:504] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
W20240503 13:56:29.786669 13252 consensus_peers.cc:485] T 9eb0d911701b4b2bb33a6b7ab59f08aa P ba0194098aa84449ae17e8a0d4c43703 -> Peer 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153): Couldn't send request to peer 61b887f0d40d497badeca5604c1f0546. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9eb0d911701b4b2bb33a6b7ab59f08aa. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:29.786710 13447 raft_consensus.cc:1261] T 9eb0d911701b4b2bb33a6b7ab59f08aa P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:56:29.786813 16161 leader_election.cc:288] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 election: Requested vote from peers 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951), 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885)
I20240503 13:56:29.787096 15625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bdb9c14caf04abca39e56f8bcf1d1d4" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d"
I20240503 13:56:29.787199 15625 raft_consensus.cc:3037] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:29.787485 15154 consensus_queue.cc:1035] T 9eb0d911701b4b2bb33a6b7ab59f08aa P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:29.788120 15625 raft_consensus.cc:2454] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 1.
I20240503 13:56:29.788414 15822 leader_election.cc:302] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:29.789175 12899 catalog_manager.cc:4830] TS 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885): tablet 97cd45c1b3b342a5aa1e3b1d24d89963 (table rebalancer_dynamic_table_new_name [id=4303f2a0523d40c090cfaaa50a92e519]) successfully deleted
I20240503 13:56:29.792845 13184 raft_consensus.cc:1261] T 9eb0d911701b4b2bb33a6b7ab59f08aa P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:56:29.793262 16229 consensus_queue.cc:1035] T 9eb0d911701b4b2bb33a6b7ab59f08aa P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:29.793370 16161 raft_consensus.cc:2781] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:29.793581 16161 raft_consensus.cc:686] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P e47105a0f46e4511b7112e571688a4c7 [term 1 LEADER]: Becoming Leader. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Running, Role: LEADER
I20240503 13:56:29.793838 16161 consensus_queue.cc:237] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:29.795531 16229 raft_consensus.cc:2932] T 9eb0d911701b4b2bb33a6b7ab59f08aa P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 61b887f0d40d497badeca5604c1f0546 (127.6.98.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:29.795827 13184 raft_consensus.cc:2932] T 9eb0d911701b4b2bb33a6b7ab59f08aa P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 61b887f0d40d497badeca5604c1f0546 (127.6.98.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:29.797708 12909 catalog_manager.cc:5484] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P e47105a0f46e4511b7112e571688a4c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e47105a0f46e4511b7112e571688a4c7 (127.6.98.72). New cstate: current_term: 1 leader_uuid: "e47105a0f46e4511b7112e571688a4c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:29.798453 12911 catalog_manager.cc:5484] T 9eb0d911701b4b2bb33a6b7ab59f08aa P ba0194098aa84449ae17e8a0d4c43703 reported cstate change: config changed from index -1 to 4, VOTER 61b887f0d40d497badeca5604c1f0546 (127.6.98.69) added. New cstate: current_term: 1 leader_uuid: "ba0194098aa84449ae17e8a0d4c43703" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:29.799229 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bdb9c14caf04abca39e56f8bcf1d1d4" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2"
I20240503 13:56:29.799353 15757 raft_consensus.cc:3037] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:29.800419 15757 raft_consensus.cc:2454] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 1.
I20240503 13:56:29.799228 15758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bdb9c14caf04abca39e56f8bcf1d1d4" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2" is_pre_election: true
I20240503 13:56:29.800768 15758 raft_consensus.cc:2362] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 1. Re-sending same reply.
I20240503 13:56:29.801338 13446 raft_consensus.cc:2932] T 9eb0d911701b4b2bb33a6b7ab59f08aa P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 61b887f0d40d497badeca5604c1f0546 (127.6.98.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:29.822217 13315 consensus_queue.cc:237] T 1b714fbd920a447f866b8d4547275675 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:29.823738 13184 raft_consensus.cc:1261] T 1b714fbd920a447f866b8d4547275675 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 13:56:29.823812 13252 consensus_peers.cc:485] T 1b714fbd920a447f866b8d4547275675 P ba0194098aa84449ae17e8a0d4c43703 -> Peer 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885): Couldn't send request to peer 76c386bef61a4bd5a907266a5118ffe2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1b714fbd920a447f866b8d4547275675. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:29.823849 13446 raft_consensus.cc:1261] T 1b714fbd920a447f866b8d4547275675 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:56:29.824144 16225 consensus_queue.cc:1035] T 1b714fbd920a447f866b8d4547275675 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:29.824328 15152 consensus_queue.cc:1035] T 1b714fbd920a447f866b8d4547275675 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:29.826020 15152 raft_consensus.cc:2932] T 1b714fbd920a447f866b8d4547275675 P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:29.826423 13446 raft_consensus.cc:2932] T 1b714fbd920a447f866b8d4547275675 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:29.828267 13184 raft_consensus.cc:2932] T 1b714fbd920a447f866b8d4547275675 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:29.829087 12911 catalog_manager.cc:5484] T 1b714fbd920a447f866b8d4547275675 P ba0194098aa84449ae17e8a0d4c43703 reported cstate change: config changed from index -1 to 4, VOTER 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71) added. New cstate: current_term: 1 leader_uuid: "ba0194098aa84449ae17e8a0d4c43703" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:29.841734 16345 raft_consensus.cc:482] T a3c4cbf779554d8b84a48de78a9c392e P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:29.841867 16345 raft_consensus.cc:504] T a3c4cbf779554d8b84a48de78a9c392e P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:29.842389 16345 leader_election.cc:288] T a3c4cbf779554d8b84a48de78a9c392e P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153)
I20240503 13:56:29.846196 15889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3c4cbf779554d8b84a48de78a9c392e" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7" is_pre_election: true
I20240503 13:56:29.846365 15889 raft_consensus.cc:2454] T a3c4cbf779554d8b84a48de78a9c392e P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 0.
I20240503 13:56:29.846678 13118 leader_election.cc:302] T a3c4cbf779554d8b84a48de78a9c392e P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ccf5a4ac450414c9d3579bd5800b770, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:29.846861 16345 raft_consensus.cc:2781] T a3c4cbf779554d8b84a48de78a9c392e P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:29.846920 16345 raft_consensus.cc:482] T a3c4cbf779554d8b84a48de78a9c392e P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:29.846966 16345 raft_consensus.cc:3037] T a3c4cbf779554d8b84a48de78a9c392e P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:29.847954 16345 raft_consensus.cc:504] T a3c4cbf779554d8b84a48de78a9c392e P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:29.848338 16345 leader_election.cc:288] T a3c4cbf779554d8b84a48de78a9c392e P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 election: Requested vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153)
I20240503 13:56:29.848572 15493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3c4cbf779554d8b84a48de78a9c392e" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546" is_pre_election: true
I20240503 13:56:29.848747 15493 raft_consensus.cc:2454] T a3c4cbf779554d8b84a48de78a9c392e P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 0.
I20240503 13:56:29.848991 15492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3c4cbf779554d8b84a48de78a9c392e" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546"
I20240503 13:56:29.849094 15492 raft_consensus.cc:3037] T a3c4cbf779554d8b84a48de78a9c392e P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:29.849999 15492 raft_consensus.cc:2454] T a3c4cbf779554d8b84a48de78a9c392e P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 1.
I20240503 13:56:29.850275 15889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3c4cbf779554d8b84a48de78a9c392e" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7"
I20240503 13:56:29.850416 15889 raft_consensus.cc:3037] T a3c4cbf779554d8b84a48de78a9c392e P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:29.850539 13121 leader_election.cc:302] T a3c4cbf779554d8b84a48de78a9c392e P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61b887f0d40d497badeca5604c1f0546, 8ccf5a4ac450414c9d3579bd5800b770; no voters:
I20240503 13:56:29.850903 16345 raft_consensus.cc:2781] T a3c4cbf779554d8b84a48de78a9c392e P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:29.850988 16345 raft_consensus.cc:686] T a3c4cbf779554d8b84a48de78a9c392e P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 LEADER]: Becoming Leader. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Running, Role: LEADER
I20240503 13:56:29.851119 16345 consensus_queue.cc:237] T a3c4cbf779554d8b84a48de78a9c392e P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:29.851374 15889 raft_consensus.cc:2454] T a3c4cbf779554d8b84a48de78a9c392e P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 1.
I20240503 13:56:29.851671 13446 consensus_queue.cc:237] T f812304076e14c2db3dc02d1ff5766fb P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:29.853567 13052 raft_consensus.cc:1261] T f812304076e14c2db3dc02d1ff5766fb P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Refusing update from remote peer 7690ee79577c45e3a2c8f8b552c0bff8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:56:29.854025 15295 consensus_queue.cc:1035] T f812304076e14c2db3dc02d1ff5766fb P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:29.854120 12905 catalog_manager.cc:5484] T a3c4cbf779554d8b84a48de78a9c392e P 8ccf5a4ac450414c9d3579bd5800b770 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66). New cstate: current_term: 1 leader_uuid: "8ccf5a4ac450414c9d3579bd5800b770" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } }
I20240503 13:56:29.856271 13184 raft_consensus.cc:1261] T f812304076e14c2db3dc02d1ff5766fb P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Refusing update from remote peer 7690ee79577c45e3a2c8f8b552c0bff8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:56:29.856933 15295 consensus_queue.cc:1035] T f812304076e14c2db3dc02d1ff5766fb P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 13:56:29.858201 13380 consensus_peers.cc:485] T f812304076e14c2db3dc02d1ff5766fb P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951): Couldn't send request to peer 009f849ee7b54c80a3f5264695047e4d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f812304076e14c2db3dc02d1ff5766fb. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:29.859145 16307 raft_consensus.cc:2932] T f812304076e14c2db3dc02d1ff5766fb P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 009f849ee7b54c80a3f5264695047e4d (127.6.98.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:29.859547 13184 raft_consensus.cc:2932] T f812304076e14c2db3dc02d1ff5766fb P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 009f849ee7b54c80a3f5264695047e4d (127.6.98.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:29.861552 13052 raft_consensus.cc:2932] T f812304076e14c2db3dc02d1ff5766fb P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 009f849ee7b54c80a3f5264695047e4d (127.6.98.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:29.863579 12909 catalog_manager.cc:5484] T f812304076e14c2db3dc02d1ff5766fb P 7690ee79577c45e3a2c8f8b552c0bff8 reported cstate change: config changed from index -1 to 4, VOTER 009f849ee7b54c80a3f5264695047e4d (127.6.98.70) added. New cstate: current_term: 1 leader_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:29.872332 12905 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:56:29.877604 15604 tablet_service.cc:1467] Processing CreateTablet for tablet 88137cb6ef4e4a6a96c5e8210fef0c96 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0a1f605d9f474a4baf4693ac6e565608]), partition=
I20240503 13:56:29.877701 15869 tablet_service.cc:1467] Processing CreateTablet for tablet 88137cb6ef4e4a6a96c5e8210fef0c96 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0a1f605d9f474a4baf4693ac6e565608]), partition=
I20240503 13:56:29.877797 15738 tablet_service.cc:1467] Processing CreateTablet for tablet 88137cb6ef4e4a6a96c5e8210fef0c96 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0a1f605d9f474a4baf4693ac6e565608]), partition=
I20240503 13:56:29.877883 15604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88137cb6ef4e4a6a96c5e8210fef0c96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:29.877947 15869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88137cb6ef4e4a6a96c5e8210fef0c96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:29.877993 15738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88137cb6ef4e4a6a96c5e8210fef0c96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:29.879982 16379 tablet_bootstrap.cc:492] T 88137cb6ef4e4a6a96c5e8210fef0c96 P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:29.880996 16379 tablet_bootstrap.cc:654] T 88137cb6ef4e4a6a96c5e8210fef0c96 P e47105a0f46e4511b7112e571688a4c7: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:29.881525 16381 tablet_bootstrap.cc:492] T 88137cb6ef4e4a6a96c5e8210fef0c96 P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:29.881945 16379 tablet_bootstrap.cc:492] T 88137cb6ef4e4a6a96c5e8210fef0c96 P e47105a0f46e4511b7112e571688a4c7: No bootstrap required, opened a new log
I20240503 13:56:29.882009 16379 ts_tablet_manager.cc:1367] T 88137cb6ef4e4a6a96c5e8210fef0c96 P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 13:56:29.882244 16381 tablet_bootstrap.cc:654] T 88137cb6ef4e4a6a96c5e8210fef0c96 P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:29.882351 16379 raft_consensus.cc:348] T 88137cb6ef4e4a6a96c5e8210fef0c96 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:29.882429 16379 raft_consensus.cc:374] T 88137cb6ef4e4a6a96c5e8210fef0c96 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:29.882448 16379 raft_consensus.cc:729] T 88137cb6ef4e4a6a96c5e8210fef0c96 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:29.882591 16379 consensus_queue.cc:260] T 88137cb6ef4e4a6a96c5e8210fef0c96 P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:29.882771 16380 tablet_bootstrap.cc:492] T 88137cb6ef4e4a6a96c5e8210fef0c96 P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:29.882855 16379 ts_tablet_manager.cc:1398] T 88137cb6ef4e4a6a96c5e8210fef0c96 P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:29.883386 16381 tablet_bootstrap.cc:492] T 88137cb6ef4e4a6a96c5e8210fef0c96 P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:29.883468 16381 ts_tablet_manager.cc:1367] T 88137cb6ef4e4a6a96c5e8210fef0c96 P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:56:29.883621 16380 tablet_bootstrap.cc:654] T 88137cb6ef4e4a6a96c5e8210fef0c96 P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:29.883911 16381 raft_consensus.cc:348] T 88137cb6ef4e4a6a96c5e8210fef0c96 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:29.883991 16381 raft_consensus.cc:374] T 88137cb6ef4e4a6a96c5e8210fef0c96 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:29.884020 16381 raft_consensus.cc:729] T 88137cb6ef4e4a6a96c5e8210fef0c96 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:29.884124 16381 consensus_queue.cc:260] T 88137cb6ef4e4a6a96c5e8210fef0c96 P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:29.884411 16381 ts_tablet_manager.cc:1398] T 88137cb6ef4e4a6a96c5e8210fef0c96 P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:29.884774 16380 tablet_bootstrap.cc:492] T 88137cb6ef4e4a6a96c5e8210fef0c96 P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:29.884863 16380 ts_tablet_manager.cc:1367] T 88137cb6ef4e4a6a96c5e8210fef0c96 P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:29.885296 16380 raft_consensus.cc:348] T 88137cb6ef4e4a6a96c5e8210fef0c96 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:29.885408 16380 raft_consensus.cc:374] T 88137cb6ef4e4a6a96c5e8210fef0c96 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:29.885452 16380 raft_consensus.cc:729] T 88137cb6ef4e4a6a96c5e8210fef0c96 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:29.885583 16380 consensus_queue.cc:260] T 88137cb6ef4e4a6a96c5e8210fef0c96 P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:29.885854 16380 ts_tablet_manager.cc:1398] T 88137cb6ef4e4a6a96c5e8210fef0c96 P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:29.889906 13052 consensus_queue.cc:237] T 7a1c36d4fd0049139c98db7af595bc62 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:29.891402 13446 raft_consensus.cc:1261] T 7a1c36d4fd0049139c98db7af595bc62 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer 60ba3ffe3b7746469834cd91c84be14e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 13:56:29.891690 12987 consensus_peers.cc:485] T 7a1c36d4fd0049139c98db7af595bc62 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7a1c36d4fd0049139c98db7af595bc62. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:29.891819 13315 raft_consensus.cc:1261] T 7a1c36d4fd0049139c98db7af595bc62 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Refusing update from remote peer 60ba3ffe3b7746469834cd91c84be14e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:56:29.891901 15958 consensus_queue.cc:1035] T 7a1c36d4fd0049139c98db7af595bc62 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:29.892244 15958 consensus_queue.cc:1035] T 7a1c36d4fd0049139c98db7af595bc62 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:29.892540 16465 ts_tablet_manager.cc:902] T b923fd3f226e409b813bb4e63325a7fc P e47105a0f46e4511b7112e571688a4c7: Initiating tablet copy from peer 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:29.893357 16465 tablet_copy_client.cc:308] T b923fd3f226e409b813bb4e63325a7fc P e47105a0f46e4511b7112e571688a4c7: tablet copy: Beginning tablet copy session from remote peer at address 127.6.98.65:34695
I20240503 13:56:29.894467 16205 raft_consensus.cc:2932] T 7a1c36d4fd0049139c98db7af595bc62 P 60ba3ffe3b7746469834cd91c84be14e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:29.894942 13446 raft_consensus.cc:2932] T 7a1c36d4fd0049139c98db7af595bc62 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:29.897536 13074 tablet_copy_service.cc:140] P 60ba3ffe3b7746469834cd91c84be14e: Received BeginTabletCopySession request for tablet b923fd3f226e409b813bb4e63325a7fc from peer e47105a0f46e4511b7112e571688a4c7 ({username='slave'} at 127.6.98.72:59759)
I20240503 13:56:29.897657 13074 tablet_copy_service.cc:161] P 60ba3ffe3b7746469834cd91c84be14e: Beginning new tablet copy session on tablet b923fd3f226e409b813bb4e63325a7fc from peer e47105a0f46e4511b7112e571688a4c7 at {username='slave'} at 127.6.98.72:59759: session id = e47105a0f46e4511b7112e571688a4c7-b923fd3f226e409b813bb4e63325a7fc
I20240503 13:56:29.897919 12911 catalog_manager.cc:5484] T 7a1c36d4fd0049139c98db7af595bc62 P 7690ee79577c45e3a2c8f8b552c0bff8 reported cstate change: config changed from index -1 to 4, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) added. New cstate: current_term: 1 leader_uuid: "60ba3ffe3b7746469834cd91c84be14e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:29.898993 13074 tablet_copy_source_session.cc:205] T b923fd3f226e409b813bb4e63325a7fc P 60ba3ffe3b7746469834cd91c84be14e: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:56:29.899232 13315 raft_consensus.cc:2932] T 7a1c36d4fd0049139c98db7af595bc62 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:29.901618 16465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b923fd3f226e409b813bb4e63325a7fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:29.904086 16465 tablet_copy_client.cc:783] T b923fd3f226e409b813bb4e63325a7fc P e47105a0f46e4511b7112e571688a4c7: tablet copy: Starting download of 0 data blocks...
I20240503 13:56:29.904304 16465 tablet_copy_client.cc:647] T b923fd3f226e409b813bb4e63325a7fc P e47105a0f46e4511b7112e571688a4c7: tablet copy: Starting download of 1 WAL segments...
I20240503 13:56:29.905982 16465 tablet_copy_client.cc:523] T b923fd3f226e409b813bb4e63325a7fc P e47105a0f46e4511b7112e571688a4c7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:56:29.907774 16465 tablet_bootstrap.cc:492] T b923fd3f226e409b813bb4e63325a7fc P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:29.908592 12909 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalance_test_table_4" } modify_external_catalogs: true
I20240503 13:56:29.908732 12909 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalance_test_table_4" } modify_external_catalogs: true
I20240503 13:56:29.911149 12909 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Sending DeleteTablet for 3 replicas of tablet b68bfa77367e4297a612610812eb8dfd
I20240503 13:56:29.911429 12909 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Sending DeleteTablet for 4 replicas of tablet b26d8338f20749ed92f5d7670e9549d5
I20240503 13:56:29.911669 12909 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Sending DeleteTablet for 3 replicas of tablet 809936cd39ce41229cc779f59c69900a
I20240503 13:56:29.913036 13423 tablet_service.cc:1514] Processing DeleteTablet for tablet b26d8338f20749ed92f5d7670e9549d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:29 UTC) from {username='slave'} at 127.0.0.1:33678
I20240503 13:56:29.913247 13295 tablet_service.cc:1514] Processing DeleteTablet for tablet b68bfa77367e4297a612610812eb8dfd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:29 UTC) from {username='slave'} at 127.0.0.1:33250
I20240503 13:56:29.913406 13427 tablet_service.cc:1514] Processing DeleteTablet for tablet 809936cd39ce41229cc779f59c69900a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:29 UTC) from {username='slave'} at 127.0.0.1:33678
I20240503 13:56:29.913544 13296 tablet_service.cc:1514] Processing DeleteTablet for tablet 809936cd39ce41229cc779f59c69900a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:29 UTC) from {username='slave'} at 127.0.0.1:33250
I20240503 13:56:29.913702 13165 tablet_service.cc:1514] Processing DeleteTablet for tablet b68bfa77367e4297a612610812eb8dfd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:29 UTC) from {username='slave'} at 127.0.0.1:41668
I20240503 13:56:29.913873 13164 tablet_service.cc:1514] Processing DeleteTablet for tablet b26d8338f20749ed92f5d7670e9549d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:29 UTC) from {username='slave'} at 127.0.0.1:41668
I20240503 13:56:29.914062 15604 tablet_service.cc:1514] Processing DeleteTablet for tablet b26d8338f20749ed92f5d7670e9549d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:29 UTC) from {username='slave'} at 127.0.0.1:60718
I20240503 13:56:29.914270 16087 tablet_replica.cc:330] T b26d8338f20749ed92f5d7670e9549d5 P 7690ee79577c45e3a2c8f8b552c0bff8: stopping tablet replica
I20240503 13:56:29.914433 16087 raft_consensus.cc:2229] T b26d8338f20749ed92f5d7670e9549d5 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:29.914611 16087 raft_consensus.cc:2258] T b26d8338f20749ed92f5d7670e9549d5 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:29.914705 13032 tablet_service.cc:1514] Processing DeleteTablet for tablet b68bfa77367e4297a612610812eb8dfd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:29 UTC) from {username='slave'} at 127.0.0.1:59458
I20240503 13:56:29.914779 16086 tablet_replica.cc:330] T b68bfa77367e4297a612610812eb8dfd P ba0194098aa84449ae17e8a0d4c43703: stopping tablet replica
I20240503 13:56:29.914847 13033 tablet_service.cc:1514] Processing DeleteTablet for tablet b26d8338f20749ed92f5d7670e9549d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:29 UTC) from {username='slave'} at 127.0.0.1:59458
I20240503 13:56:29.914902 16086 raft_consensus.cc:2229] T b68bfa77367e4297a612610812eb8dfd P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:56:29.914961 16085 tablet_replica.cc:330] T b68bfa77367e4297a612610812eb8dfd P 60ba3ffe3b7746469834cd91c84be14e: stopping tablet replica
I20240503 13:56:29.915072 16085 raft_consensus.cc:2229] T b68bfa77367e4297a612610812eb8dfd P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:29.915089 16086 raft_consensus.cc:2258] T b68bfa77367e4297a612610812eb8dfd P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:29.915222 16085 raft_consensus.cc:2258] T b68bfa77367e4297a612610812eb8dfd P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:29.915596 16086 ts_tablet_manager.cc:1875] T b68bfa77367e4297a612610812eb8dfd P ba0194098aa84449ae17e8a0d4c43703: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:29.915683 16085 ts_tablet_manager.cc:1875] T b68bfa77367e4297a612610812eb8dfd P 60ba3ffe3b7746469834cd91c84be14e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:29.917898 16092 tablet_replica.cc:330] T b68bfa77367e4297a612610812eb8dfd P 8ccf5a4ac450414c9d3579bd5800b770: stopping tablet replica
I20240503 13:56:29.917974 16085 ts_tablet_manager.cc:1888] T b68bfa77367e4297a612610812eb8dfd P 60ba3ffe3b7746469834cd91c84be14e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:56:29.918030 16085 log.cc:1196] T b68bfa77367e4297a612610812eb8dfd P 60ba3ffe3b7746469834cd91c84be14e: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal/wals/b68bfa77367e4297a612610812eb8dfd
I20240503 13:56:29.918048 16092 raft_consensus.cc:2229] T b68bfa77367e4297a612610812eb8dfd P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:29.918176 16092 raft_consensus.cc:2258] T b68bfa77367e4297a612610812eb8dfd P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:29.918267 16085 ts_tablet_manager.cc:1909] T b68bfa77367e4297a612610812eb8dfd P 60ba3ffe3b7746469834cd91c84be14e: Deleting consensus metadata
I20240503 13:56:29.918645 16092 ts_tablet_manager.cc:1875] T b68bfa77367e4297a612610812eb8dfd P 8ccf5a4ac450414c9d3579bd5800b770: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:29.919188 12897 catalog_manager.cc:4830] TS 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695): tablet b68bfa77367e4297a612610812eb8dfd (table rebalance_test_table_4 [id=6e1f1c4458104a159283843ed95fe485]) successfully deleted
I20240503 13:56:29.919371 16085 tablet_replica.cc:330] T b26d8338f20749ed92f5d7670e9549d5 P 60ba3ffe3b7746469834cd91c84be14e: stopping tablet replica
I20240503 13:56:29.919476 16085 raft_consensus.cc:2229] T b26d8338f20749ed92f5d7670e9549d5 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:29.919621 16085 raft_consensus.cc:2258] T b26d8338f20749ed92f5d7670e9549d5 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:29.920218 16085 ts_tablet_manager.cc:1875] T b26d8338f20749ed92f5d7670e9549d5 P 60ba3ffe3b7746469834cd91c84be14e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:29.920802 13034 tablet_service.cc:1514] Processing DeleteTablet for tablet 809936cd39ce41229cc779f59c69900a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:29 UTC) from {username='slave'} at 127.0.0.1:59458
I20240503 13:56:29.922278 16085 ts_tablet_manager.cc:1888] T b26d8338f20749ed92f5d7670e9549d5 P 60ba3ffe3b7746469834cd91c84be14e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:29.922333 16085 log.cc:1196] T b26d8338f20749ed92f5d7670e9549d5 P 60ba3ffe3b7746469834cd91c84be14e: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal/wals/b26d8338f20749ed92f5d7670e9549d5
I20240503 13:56:29.922631 16085 ts_tablet_manager.cc:1909] T b26d8338f20749ed92f5d7670e9549d5 P 60ba3ffe3b7746469834cd91c84be14e: Deleting consensus metadata
I20240503 13:56:29.923287 12897 catalog_manager.cc:4830] TS 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695): tablet b26d8338f20749ed92f5d7670e9549d5 (table rebalance_test_table_4 [id=6e1f1c4458104a159283843ed95fe485]) successfully deleted
W20240503 13:56:29.923321 12896 catalog_manager.cc:4797] TS 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951): delete failed for tablet b26d8338f20749ed92f5d7670e9549d5 because the tablet was not found. No further retry: Not found: Tablet not found: b26d8338f20749ed92f5d7670e9549d5
I20240503 13:56:29.925100 16087 ts_tablet_manager.cc:1875] T b26d8338f20749ed92f5d7670e9549d5 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:29.926169 16085 tablet_replica.cc:330] T 809936cd39ce41229cc779f59c69900a P 60ba3ffe3b7746469834cd91c84be14e: stopping tablet replica
I20240503 13:56:29.926302 16085 raft_consensus.cc:2229] T 809936cd39ce41229cc779f59c69900a P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:29.926400 16085 raft_consensus.cc:2258] T 809936cd39ce41229cc779f59c69900a P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:29.926844 16085 ts_tablet_manager.cc:1875] T 809936cd39ce41229cc779f59c69900a P 60ba3ffe3b7746469834cd91c84be14e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:29.927120 16087 ts_tablet_manager.cc:1888] T b26d8338f20749ed92f5d7670e9549d5 P 7690ee79577c45e3a2c8f8b552c0bff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:29.927182 16087 log.cc:1196] T b26d8338f20749ed92f5d7670e9549d5 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal/wals/b26d8338f20749ed92f5d7670e9549d5
I20240503 13:56:29.927417 16087 ts_tablet_manager.cc:1909] T b26d8338f20749ed92f5d7670e9549d5 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting consensus metadata
I20240503 13:56:29.928263 16087 tablet_replica.cc:330] T 809936cd39ce41229cc779f59c69900a P 7690ee79577c45e3a2c8f8b552c0bff8: stopping tablet replica
I20240503 13:56:29.928313 16092 ts_tablet_manager.cc:1888] T b68bfa77367e4297a612610812eb8dfd P 8ccf5a4ac450414c9d3579bd5800b770: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:56:29.928400 16092 log.cc:1196] T b68bfa77367e4297a612610812eb8dfd P 8ccf5a4ac450414c9d3579bd5800b770: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/wal/wals/b68bfa77367e4297a612610812eb8dfd
I20240503 13:56:29.928376 12896 catalog_manager.cc:4830] TS 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109): tablet b26d8338f20749ed92f5d7670e9549d5 (table rebalance_test_table_4 [id=6e1f1c4458104a159283843ed95fe485]) successfully deleted
I20240503 13:56:29.928668 16087 raft_consensus.cc:2229] T 809936cd39ce41229cc779f59c69900a P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:29.928874 16087 raft_consensus.cc:2258] T 809936cd39ce41229cc779f59c69900a P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:29.928900 16086 ts_tablet_manager.cc:1888] T b68bfa77367e4297a612610812eb8dfd P ba0194098aa84449ae17e8a0d4c43703: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:56:29.928989 16086 log.cc:1196] T b68bfa77367e4297a612610812eb8dfd P ba0194098aa84449ae17e8a0d4c43703: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/wal/wals/b68bfa77367e4297a612610812eb8dfd
I20240503 13:56:29.929222 16086 ts_tablet_manager.cc:1909] T b68bfa77367e4297a612610812eb8dfd P ba0194098aa84449ae17e8a0d4c43703: Deleting consensus metadata
I20240503 13:56:29.929394 16087 ts_tablet_manager.cc:1875] T 809936cd39ce41229cc779f59c69900a P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:29.928705 16092 ts_tablet_manager.cc:1909] T b68bfa77367e4297a612610812eb8dfd P 8ccf5a4ac450414c9d3579bd5800b770: Deleting consensus metadata
I20240503 13:56:29.930706 12899 catalog_manager.cc:4830] TS 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387): tablet b68bfa77367e4297a612610812eb8dfd (table rebalance_test_table_4 [id=6e1f1c4458104a159283843ed95fe485]) successfully deleted
I20240503 13:56:29.931227 12899 catalog_manager.cc:4830] TS ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279): tablet b68bfa77367e4297a612610812eb8dfd (table rebalance_test_table_4 [id=6e1f1c4458104a159283843ed95fe485]) successfully deleted
I20240503 13:56:29.931463 16086 tablet_replica.cc:330] T 809936cd39ce41229cc779f59c69900a P ba0194098aa84449ae17e8a0d4c43703: stopping tablet replica
I20240503 13:56:29.931620 16092 tablet_replica.cc:330] T b26d8338f20749ed92f5d7670e9549d5 P 8ccf5a4ac450414c9d3579bd5800b770: stopping tablet replica
I20240503 13:56:29.931707 16087 ts_tablet_manager.cc:1888] T 809936cd39ce41229cc779f59c69900a P 7690ee79577c45e3a2c8f8b552c0bff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:56:29.931774 16087 log.cc:1196] T 809936cd39ce41229cc779f59c69900a P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal/wals/809936cd39ce41229cc779f59c69900a
I20240503 13:56:29.931936 16086 raft_consensus.cc:2229] T 809936cd39ce41229cc779f59c69900a P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:56:29.931991 16087 ts_tablet_manager.cc:1909] T 809936cd39ce41229cc779f59c69900a P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting consensus metadata
I20240503 13:56:29.932138 16092 raft_consensus.cc:2229] T b26d8338f20749ed92f5d7670e9549d5 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:56:29.932359 16086 raft_consensus.cc:2258] T 809936cd39ce41229cc779f59c69900a P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:29.932706 16092 raft_consensus.cc:2258] T b26d8338f20749ed92f5d7670e9549d5 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:29.933027 12896 catalog_manager.cc:4830] TS 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109): tablet 809936cd39ce41229cc779f59c69900a (table rebalance_test_table_4 [id=6e1f1c4458104a159283843ed95fe485]) successfully deleted
I20240503 13:56:29.933653 16085 ts_tablet_manager.cc:1888] T 809936cd39ce41229cc779f59c69900a P 60ba3ffe3b7746469834cd91c84be14e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:56:29.933710 16085 log.cc:1196] T 809936cd39ce41229cc779f59c69900a P 60ba3ffe3b7746469834cd91c84be14e: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal/wals/809936cd39ce41229cc779f59c69900a
I20240503 13:56:29.933977 16085 ts_tablet_manager.cc:1909] T 809936cd39ce41229cc779f59c69900a P 60ba3ffe3b7746469834cd91c84be14e: Deleting consensus metadata
I20240503 13:56:29.934835 12897 catalog_manager.cc:4830] TS 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695): tablet 809936cd39ce41229cc779f59c69900a (table rebalance_test_table_4 [id=6e1f1c4458104a159283843ed95fe485]) successfully deleted
I20240503 13:56:29.936228 16092 ts_tablet_manager.cc:1875] T b26d8338f20749ed92f5d7670e9549d5 P 8ccf5a4ac450414c9d3579bd5800b770: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:29.936638 16086 ts_tablet_manager.cc:1875] T 809936cd39ce41229cc779f59c69900a P ba0194098aa84449ae17e8a0d4c43703: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:29.938213 16484 ts_tablet_manager.cc:902] T 81d47f705c0b4128a84e4bf7d274a659 P 76c386bef61a4bd5a907266a5118ffe2: Initiating tablet copy from peer 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:29.939832 16484 tablet_copy_client.cc:308] T 81d47f705c0b4128a84e4bf7d274a659 P 76c386bef61a4bd5a907266a5118ffe2: tablet copy: Beginning tablet copy session from remote peer at address 127.6.98.68:36109
I20240503 13:56:29.940912 13467 tablet_copy_service.cc:140] P 7690ee79577c45e3a2c8f8b552c0bff8: Received BeginTabletCopySession request for tablet 81d47f705c0b4128a84e4bf7d274a659 from peer 76c386bef61a4bd5a907266a5118ffe2 ({username='slave'} at 127.6.98.71:41539)
I20240503 13:56:29.941040 13467 tablet_copy_service.cc:161] P 7690ee79577c45e3a2c8f8b552c0bff8: Beginning new tablet copy session on tablet 81d47f705c0b4128a84e4bf7d274a659 from peer 76c386bef61a4bd5a907266a5118ffe2 at {username='slave'} at 127.6.98.71:41539: session id = 76c386bef61a4bd5a907266a5118ffe2-81d47f705c0b4128a84e4bf7d274a659
I20240503 13:56:29.942209 13467 tablet_copy_source_session.cc:205] T 81d47f705c0b4128a84e4bf7d274a659 P 7690ee79577c45e3a2c8f8b552c0bff8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:56:29.942708 16465 tablet_bootstrap.cc:492] T b923fd3f226e409b813bb4e63325a7fc P e47105a0f46e4511b7112e571688a4c7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:56:29.943027 16465 tablet_bootstrap.cc:492] T b923fd3f226e409b813bb4e63325a7fc P e47105a0f46e4511b7112e571688a4c7: Bootstrap complete.
I20240503 13:56:29.943306 16465 ts_tablet_manager.cc:1367] T b923fd3f226e409b813bb4e63325a7fc P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.036s user 0.029s sys 0.000s
I20240503 13:56:29.943840 16465 raft_consensus.cc:348] T b923fd3f226e409b813bb4e63325a7fc P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:29.944026 16465 raft_consensus.cc:729] T b923fd3f226e409b813bb4e63325a7fc P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:29.944073 16484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81d47f705c0b4128a84e4bf7d274a659. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:29.944298 16465 consensus_queue.cc:260] T b923fd3f226e409b813bb4e63325a7fc P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:29.945279 16465 ts_tablet_manager.cc:1398] T b923fd3f226e409b813bb4e63325a7fc P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:29.945885 13074 tablet_copy_service.cc:342] P 60ba3ffe3b7746469834cd91c84be14e: Request end of tablet copy session e47105a0f46e4511b7112e571688a4c7-b923fd3f226e409b813bb4e63325a7fc received from {username='slave'} at 127.6.98.72:59759
I20240503 13:56:29.945974 13074 tablet_copy_service.cc:433] P 60ba3ffe3b7746469834cd91c84be14e: ending tablet copy session e47105a0f46e4511b7112e571688a4c7-b923fd3f226e409b813bb4e63325a7fc on tablet b923fd3f226e409b813bb4e63325a7fc with peer e47105a0f46e4511b7112e571688a4c7
I20240503 13:56:29.947995 16484 tablet_copy_client.cc:783] T 81d47f705c0b4128a84e4bf7d274a659 P 76c386bef61a4bd5a907266a5118ffe2: tablet copy: Starting download of 0 data blocks...
I20240503 13:56:29.948235 16484 tablet_copy_client.cc:647] T 81d47f705c0b4128a84e4bf7d274a659 P 76c386bef61a4bd5a907266a5118ffe2: tablet copy: Starting download of 1 WAL segments...
I20240503 13:56:29.949290 16092 ts_tablet_manager.cc:1888] T b26d8338f20749ed92f5d7670e9549d5 P 8ccf5a4ac450414c9d3579bd5800b770: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:29.949371 16092 log.cc:1196] T b26d8338f20749ed92f5d7670e9549d5 P 8ccf5a4ac450414c9d3579bd5800b770: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/wal/wals/b26d8338f20749ed92f5d7670e9549d5
I20240503 13:56:29.949669 16092 ts_tablet_manager.cc:1909] T b26d8338f20749ed92f5d7670e9549d5 P 8ccf5a4ac450414c9d3579bd5800b770: Deleting consensus metadata
I20240503 13:56:29.949702 16086 ts_tablet_manager.cc:1888] T 809936cd39ce41229cc779f59c69900a P ba0194098aa84449ae17e8a0d4c43703: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:56:29.949801 16086 log.cc:1196] T 809936cd39ce41229cc779f59c69900a P ba0194098aa84449ae17e8a0d4c43703: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/wal/wals/809936cd39ce41229cc779f59c69900a
I20240503 13:56:29.950050 16086 ts_tablet_manager.cc:1909] T 809936cd39ce41229cc779f59c69900a P ba0194098aa84449ae17e8a0d4c43703: Deleting consensus metadata
I20240503 13:56:29.950574 12899 catalog_manager.cc:4830] TS 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387): tablet b26d8338f20749ed92f5d7670e9549d5 (table rebalance_test_table_4 [id=6e1f1c4458104a159283843ed95fe485]) successfully deleted
I20240503 13:56:29.950753 12899 catalog_manager.cc:4830] TS ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279): tablet 809936cd39ce41229cc779f59c69900a (table rebalance_test_table_4 [id=6e1f1c4458104a159283843ed95fe485]) successfully deleted
I20240503 13:56:29.952927 16484 tablet_copy_client.cc:523] T 81d47f705c0b4128a84e4bf7d274a659 P 76c386bef61a4bd5a907266a5118ffe2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:56:29.953143 13052 consensus_queue.cc:237] T 05917bf47aea42d397ccae57c3f29416 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:29.954505 13446 raft_consensus.cc:1261] T 05917bf47aea42d397ccae57c3f29416 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer 60ba3ffe3b7746469834cd91c84be14e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:56:29.954638 16484 tablet_bootstrap.cc:492] T 81d47f705c0b4128a84e4bf7d274a659 P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
W20240503 13:56:29.954885 12990 consensus_peers.cc:485] T 05917bf47aea42d397ccae57c3f29416 P 60ba3ffe3b7746469834cd91c84be14e -> Peer 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885): Couldn't send request to peer 76c386bef61a4bd5a907266a5118ffe2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 05917bf47aea42d397ccae57c3f29416. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:29.954999 16472 consensus_queue.cc:1035] T 05917bf47aea42d397ccae57c3f29416 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:29.955719 13315 raft_consensus.cc:1261] T 05917bf47aea42d397ccae57c3f29416 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Refusing update from remote peer 60ba3ffe3b7746469834cd91c84be14e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:56:29.956244 16472 consensus_queue.cc:1035] T 05917bf47aea42d397ccae57c3f29416 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:29.956607 15956 raft_consensus.cc:482] T 88137cb6ef4e4a6a96c5e8210fef0c96 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:29.956694 15956 raft_consensus.cc:504] T 88137cb6ef4e4a6a96c5e8210fef0c96 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:29.957041 15956 leader_election.cc:288] T 88137cb6ef4e4a6a96c5e8210fef0c96 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:29.957502 15625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88137cb6ef4e4a6a96c5e8210fef0c96" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d" is_pre_election: true
I20240503 13:56:29.957640 15625 raft_consensus.cc:2454] T 88137cb6ef4e4a6a96c5e8210fef0c96 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 0.
I20240503 13:56:29.957911 15889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88137cb6ef4e4a6a96c5e8210fef0c96" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7" is_pre_election: true
I20240503 13:56:29.957974 15691 leader_election.cc:302] T 88137cb6ef4e4a6a96c5e8210fef0c96 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 76c386bef61a4bd5a907266a5118ffe2; no voters:
I20240503 13:56:29.958065 15889 raft_consensus.cc:2454] T 88137cb6ef4e4a6a96c5e8210fef0c96 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 0.
I20240503 13:56:29.958139 15956 raft_consensus.cc:2781] T 88137cb6ef4e4a6a96c5e8210fef0c96 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:29.958190 15956 raft_consensus.cc:482] T 88137cb6ef4e4a6a96c5e8210fef0c96 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:29.958230 15956 raft_consensus.cc:3037] T 88137cb6ef4e4a6a96c5e8210fef0c96 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:29.958901 15332 raft_consensus.cc:2932] T 05917bf47aea42d397ccae57c3f29416 P 60ba3ffe3b7746469834cd91c84be14e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:29.959215 15956 raft_consensus.cc:504] T 88137cb6ef4e4a6a96c5e8210fef0c96 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:29.959630 15956 leader_election.cc:288] T 88137cb6ef4e4a6a96c5e8210fef0c96 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 election: Requested vote from peers 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:29.960899 13446 raft_consensus.cc:2932] T 05917bf47aea42d397ccae57c3f29416 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:29.962118 12909 catalog_manager.cc:5484] T 05917bf47aea42d397ccae57c3f29416 P 60ba3ffe3b7746469834cd91c84be14e reported cstate change: config changed from index -1 to 4, VOTER 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71) added. New cstate: current_term: 1 leader_uuid: "60ba3ffe3b7746469834cd91c84be14e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:29.965283 13314 raft_consensus.cc:2932] T 05917bf47aea42d397ccae57c3f29416 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:29.965493 15889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88137cb6ef4e4a6a96c5e8210fef0c96" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7"
I20240503 13:56:29.965613 15889 raft_consensus.cc:3037] T 88137cb6ef4e4a6a96c5e8210fef0c96 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:29.965584 15625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88137cb6ef4e4a6a96c5e8210fef0c96" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d"
I20240503 13:56:29.965700 15625 raft_consensus.cc:3037] T 88137cb6ef4e4a6a96c5e8210fef0c96 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:29.966552 15889 raft_consensus.cc:2454] T 88137cb6ef4e4a6a96c5e8210fef0c96 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 1.
I20240503 13:56:29.966579 15625 raft_consensus.cc:2454] T 88137cb6ef4e4a6a96c5e8210fef0c96 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 1.
I20240503 13:56:29.966804 16161 raft_consensus.cc:482] T 0f9224c2e3b946d993a5fa5c95a1d681 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:29.966882 16161 raft_consensus.cc:504] T 0f9224c2e3b946d993a5fa5c95a1d681 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:29.967221 16161 leader_election.cc:288] T 0f9224c2e3b946d993a5fa5c95a1d681 P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885), 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951)
I20240503 13:56:29.967520 15758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f9224c2e3b946d993a5fa5c95a1d681" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2" is_pre_election: true
I20240503 13:56:29.967638 15758 raft_consensus.cc:2454] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 0.
I20240503 13:56:29.967916 15825 leader_election.cc:302] T 0f9224c2e3b946d993a5fa5c95a1d681 P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76c386bef61a4bd5a907266a5118ffe2, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:29.968391 16161 raft_consensus.cc:2781] T 0f9224c2e3b946d993a5fa5c95a1d681 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:29.968438 16161 raft_consensus.cc:482] T 0f9224c2e3b946d993a5fa5c95a1d681 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:29.968462 16161 raft_consensus.cc:3037] T 0f9224c2e3b946d993a5fa5c95a1d681 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:29.969331 16161 raft_consensus.cc:504] T 0f9224c2e3b946d993a5fa5c95a1d681 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:29.969807 16161 leader_election.cc:288] T 0f9224c2e3b946d993a5fa5c95a1d681 P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 election: Requested vote from peers 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885), 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951)
I20240503 13:56:29.970168 15624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f9224c2e3b946d993a5fa5c95a1d681" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d"
I20240503 13:56:29.970283 15624 raft_consensus.cc:3037] T 0f9224c2e3b946d993a5fa5c95a1d681 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:29.970353 15625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f9224c2e3b946d993a5fa5c95a1d681" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d" is_pre_election: true
I20240503 13:56:29.971274 15624 raft_consensus.cc:2454] T 0f9224c2e3b946d993a5fa5c95a1d681 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 1.
I20240503 13:56:29.971616 15822 leader_election.cc:302] T 0f9224c2e3b946d993a5fa5c95a1d681 P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:29.971778 16161 raft_consensus.cc:2781] T 0f9224c2e3b946d993a5fa5c95a1d681 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:29.971866 16161 raft_consensus.cc:686] T 0f9224c2e3b946d993a5fa5c95a1d681 P e47105a0f46e4511b7112e571688a4c7 [term 1 LEADER]: Becoming Leader. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Running, Role: LEADER
I20240503 13:56:29.971881 15691 leader_election.cc:302] T 88137cb6ef4e4a6a96c5e8210fef0c96 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 76c386bef61a4bd5a907266a5118ffe2; no voters:
I20240503 13:56:29.971988 16161 consensus_queue.cc:237] T 0f9224c2e3b946d993a5fa5c95a1d681 P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:29.972151 15956 raft_consensus.cc:2781] T 88137cb6ef4e4a6a96c5e8210fef0c96 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:29.972215 15758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f9224c2e3b946d993a5fa5c95a1d681" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2"
I20240503 13:56:29.972330 15758 raft_consensus.cc:3037] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:29.973270 15758 raft_consensus.cc:2454] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 1.
I20240503 13:56:29.972225 15956 raft_consensus.cc:686] T 88137cb6ef4e4a6a96c5e8210fef0c96 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 LEADER]: Becoming Leader. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Running, Role: LEADER
I20240503 13:56:29.973555 15956 consensus_queue.cc:237] T 88137cb6ef4e4a6a96c5e8210fef0c96 P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:29.973631 12905 catalog_manager.cc:5484] T 0f9224c2e3b946d993a5fa5c95a1d681 P e47105a0f46e4511b7112e571688a4c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e47105a0f46e4511b7112e571688a4c7 (127.6.98.72). New cstate: current_term: 1 leader_uuid: "e47105a0f46e4511b7112e571688a4c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:29.975211 12909 catalog_manager.cc:5484] T 88137cb6ef4e4a6a96c5e8210fef0c96 P 76c386bef61a4bd5a907266a5118ffe2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71). New cstate: current_term: 1 leader_uuid: "76c386bef61a4bd5a907266a5118ffe2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:29.992419 16484 tablet_bootstrap.cc:492] T 81d47f705c0b4128a84e4bf7d274a659 P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:56:29.992784 16484 tablet_bootstrap.cc:492] T 81d47f705c0b4128a84e4bf7d274a659 P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap complete.
I20240503 13:56:29.993010 16484 ts_tablet_manager.cc:1367] T 81d47f705c0b4128a84e4bf7d274a659 P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.038s user 0.023s sys 0.007s
I20240503 13:56:29.993443 16484 raft_consensus.cc:348] T 81d47f705c0b4128a84e4bf7d274a659 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:29.993609 16484 raft_consensus.cc:729] T 81d47f705c0b4128a84e4bf7d274a659 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:29.993790 16484 consensus_queue.cc:260] T 81d47f705c0b4128a84e4bf7d274a659 P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:29.994161 16484 ts_tablet_manager.cc:1398] T 81d47f705c0b4128a84e4bf7d274a659 P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:29.994822 13467 tablet_copy_service.cc:342] P 7690ee79577c45e3a2c8f8b552c0bff8: Request end of tablet copy session 76c386bef61a4bd5a907266a5118ffe2-81d47f705c0b4128a84e4bf7d274a659 received from {username='slave'} at 127.6.98.71:41539
I20240503 13:56:29.994908 13467 tablet_copy_service.cc:433] P 7690ee79577c45e3a2c8f8b552c0bff8: ending tablet copy session 76c386bef61a4bd5a907266a5118ffe2-81d47f705c0b4128a84e4bf7d274a659 on tablet 81d47f705c0b4128a84e4bf7d274a659 with peer 76c386bef61a4bd5a907266a5118ffe2
I20240503 13:56:29.997689 13315 consensus_queue.cc:237] T 3bb148fcf60f410089420a9499f08086 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:29.999104 13184 raft_consensus.cc:1261] T 3bb148fcf60f410089420a9499f08086 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:56:29.999114 13446 raft_consensus.cc:1261] T 3bb148fcf60f410089420a9499f08086 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 13:56:29.999318 13252 consensus_peers.cc:485] T 3bb148fcf60f410089420a9499f08086 P ba0194098aa84449ae17e8a0d4c43703 -> Peer 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153): Couldn't send request to peer 61b887f0d40d497badeca5604c1f0546. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3bb148fcf60f410089420a9499f08086. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:29.999586 16225 consensus_queue.cc:1035] T 3bb148fcf60f410089420a9499f08086 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:29.999754 15152 consensus_queue.cc:1035] T 3bb148fcf60f410089420a9499f08086 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:30.001580 16225 raft_consensus.cc:2932] T 3bb148fcf60f410089420a9499f08086 P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 61b887f0d40d497badeca5604c1f0546 (127.6.98.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:30.001988 13446 raft_consensus.cc:2932] T 3bb148fcf60f410089420a9499f08086 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 61b887f0d40d497badeca5604c1f0546 (127.6.98.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:30.003719 13184 raft_consensus.cc:2932] T 3bb148fcf60f410089420a9499f08086 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 61b887f0d40d497badeca5604c1f0546 (127.6.98.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:30.004354 12909 catalog_manager.cc:5484] T 3bb148fcf60f410089420a9499f08086 P ba0194098aa84449ae17e8a0d4c43703 reported cstate change: config changed from index -1 to 4, VOTER 61b887f0d40d497badeca5604c1f0546 (127.6.98.69) added. New cstate: current_term: 1 leader_uuid: "ba0194098aa84449ae17e8a0d4c43703" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:30.030031 13447 consensus_queue.cc:237] T 866a258ecb2e4685908d5382cb594959 P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:30.031735 13052 raft_consensus.cc:1261] T 866a258ecb2e4685908d5382cb594959 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Refusing update from remote peer 7690ee79577c45e3a2c8f8b552c0bff8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:56:30.031946 13184 raft_consensus.cc:1261] T 866a258ecb2e4685908d5382cb594959 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Refusing update from remote peer 7690ee79577c45e3a2c8f8b552c0bff8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 13:56:30.031975 13380 consensus_peers.cc:485] T 866a258ecb2e4685908d5382cb594959 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951): Couldn't send request to peer 009f849ee7b54c80a3f5264695047e4d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 866a258ecb2e4685908d5382cb594959. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:30.032317 16307 consensus_queue.cc:1035] T 866a258ecb2e4685908d5382cb594959 P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:30.032768 16307 consensus_queue.cc:1035] T 866a258ecb2e4685908d5382cb594959 P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:30.034755 16307 raft_consensus.cc:2932] T 866a258ecb2e4685908d5382cb594959 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 009f849ee7b54c80a3f5264695047e4d (127.6.98.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:30.035044 16465 ts_tablet_manager.cc:902] T 4bd42569fd3b4b39b0bd489acebbad10 P e47105a0f46e4511b7112e571688a4c7: Initiating tablet copy from peer 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:30.035519 16465 tablet_copy_client.cc:308] T 4bd42569fd3b4b39b0bd489acebbad10 P e47105a0f46e4511b7112e571688a4c7: tablet copy: Beginning tablet copy session from remote peer at address 127.6.98.65:34695
I20240503 13:56:30.035393 13052 raft_consensus.cc:2932] T 866a258ecb2e4685908d5382cb594959 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 009f849ee7b54c80a3f5264695047e4d (127.6.98.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:30.035964 13074 tablet_copy_service.cc:140] P 60ba3ffe3b7746469834cd91c84be14e: Received BeginTabletCopySession request for tablet 4bd42569fd3b4b39b0bd489acebbad10 from peer e47105a0f46e4511b7112e571688a4c7 ({username='slave'} at 127.6.98.72:59759)
I20240503 13:56:30.036041 13074 tablet_copy_service.cc:161] P 60ba3ffe3b7746469834cd91c84be14e: Beginning new tablet copy session on tablet 4bd42569fd3b4b39b0bd489acebbad10 from peer e47105a0f46e4511b7112e571688a4c7 at {username='slave'} at 127.6.98.72:59759: session id = e47105a0f46e4511b7112e571688a4c7-4bd42569fd3b4b39b0bd489acebbad10
I20240503 13:56:30.037786 13074 tablet_copy_source_session.cc:205] T 4bd42569fd3b4b39b0bd489acebbad10 P 60ba3ffe3b7746469834cd91c84be14e: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:56:30.038853 16465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bd42569fd3b4b39b0bd489acebbad10. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:30.038765 12911 catalog_manager.cc:5484] T 866a258ecb2e4685908d5382cb594959 P 7690ee79577c45e3a2c8f8b552c0bff8 reported cstate change: config changed from index -1 to 4, VOTER 009f849ee7b54c80a3f5264695047e4d (127.6.98.70) added. New cstate: current_term: 1 leader_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:30.040952 16465 tablet_copy_client.cc:783] T 4bd42569fd3b4b39b0bd489acebbad10 P e47105a0f46e4511b7112e571688a4c7: tablet copy: Starting download of 0 data blocks...
I20240503 13:56:30.041111 16465 tablet_copy_client.cc:647] T 4bd42569fd3b4b39b0bd489acebbad10 P e47105a0f46e4511b7112e571688a4c7: tablet copy: Starting download of 1 WAL segments...
I20240503 13:56:30.041311 13184 raft_consensus.cc:2932] T 866a258ecb2e4685908d5382cb594959 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 009f849ee7b54c80a3f5264695047e4d (127.6.98.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:30.042402 16465 tablet_copy_client.cc:523] T 4bd42569fd3b4b39b0bd489acebbad10 P e47105a0f46e4511b7112e571688a4c7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:56:30.043453 16465 tablet_bootstrap.cc:492] T 4bd42569fd3b4b39b0bd489acebbad10 P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:30.047264 16540 ts_tablet_manager.cc:902] T fd6a3d908840456f82c346c4bf1d8160 P 61b887f0d40d497badeca5604c1f0546: Initiating tablet copy from peer ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:30.049293 16540 tablet_copy_client.cc:308] T fd6a3d908840456f82c346c4bf1d8160 P 61b887f0d40d497badeca5604c1f0546: tablet copy: Beginning tablet copy session from remote peer at address 127.6.98.67:42279
I20240503 13:56:30.058295 13336 tablet_copy_service.cc:140] P ba0194098aa84449ae17e8a0d4c43703: Received BeginTabletCopySession request for tablet fd6a3d908840456f82c346c4bf1d8160 from peer 61b887f0d40d497badeca5604c1f0546 ({username='slave'} at 127.6.98.69:36581)
I20240503 13:56:30.058425 13336 tablet_copy_service.cc:161] P ba0194098aa84449ae17e8a0d4c43703: Beginning new tablet copy session on tablet fd6a3d908840456f82c346c4bf1d8160 from peer 61b887f0d40d497badeca5604c1f0546 at {username='slave'} at 127.6.98.69:36581: session id = 61b887f0d40d497badeca5604c1f0546-fd6a3d908840456f82c346c4bf1d8160
I20240503 13:56:30.059572 13336 tablet_copy_source_session.cc:205] T fd6a3d908840456f82c346c4bf1d8160 P ba0194098aa84449ae17e8a0d4c43703: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:56:30.061239 16558 ts_tablet_manager.cc:902] T 4647faf6a457415d8ae2b6691f452eab P 009f849ee7b54c80a3f5264695047e4d: Initiating tablet copy from peer ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:30.061374 16540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd6a3d908840456f82c346c4bf1d8160. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:30.062048 16558 tablet_copy_client.cc:308] T 4647faf6a457415d8ae2b6691f452eab P 009f849ee7b54c80a3f5264695047e4d: tablet copy: Beginning tablet copy session from remote peer at address 127.6.98.67:42279
I20240503 13:56:30.063977 16540 tablet_copy_client.cc:783] T fd6a3d908840456f82c346c4bf1d8160 P 61b887f0d40d497badeca5604c1f0546: tablet copy: Starting download of 0 data blocks...
I20240503 13:56:30.064179 16540 tablet_copy_client.cc:647] T fd6a3d908840456f82c346c4bf1d8160 P 61b887f0d40d497badeca5604c1f0546: tablet copy: Starting download of 1 WAL segments...
I20240503 13:56:30.066530 16540 tablet_copy_client.cc:523] T fd6a3d908840456f82c346c4bf1d8160 P 61b887f0d40d497badeca5604c1f0546: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:56:30.069590 16540 tablet_bootstrap.cc:492] T fd6a3d908840456f82c346c4bf1d8160 P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:30.070590 13336 tablet_copy_service.cc:140] P ba0194098aa84449ae17e8a0d4c43703: Received BeginTabletCopySession request for tablet 4647faf6a457415d8ae2b6691f452eab from peer 009f849ee7b54c80a3f5264695047e4d ({username='slave'} at 127.6.98.70:35625)
I20240503 13:56:30.070691 13336 tablet_copy_service.cc:161] P ba0194098aa84449ae17e8a0d4c43703: Beginning new tablet copy session on tablet 4647faf6a457415d8ae2b6691f452eab from peer 009f849ee7b54c80a3f5264695047e4d at {username='slave'} at 127.6.98.70:35625: session id = 009f849ee7b54c80a3f5264695047e4d-4647faf6a457415d8ae2b6691f452eab
I20240503 13:56:30.071396 13336 tablet_copy_source_session.cc:205] T 4647faf6a457415d8ae2b6691f452eab P ba0194098aa84449ae17e8a0d4c43703: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:56:30.072737 16558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4647faf6a457415d8ae2b6691f452eab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:30.074965 16558 tablet_copy_client.cc:783] T 4647faf6a457415d8ae2b6691f452eab P 009f849ee7b54c80a3f5264695047e4d: tablet copy: Starting download of 0 data blocks...
I20240503 13:56:30.075140 16558 tablet_copy_client.cc:647] T 4647faf6a457415d8ae2b6691f452eab P 009f849ee7b54c80a3f5264695047e4d: tablet copy: Starting download of 1 WAL segments...
I20240503 13:56:30.076217 16465 tablet_bootstrap.cc:492] T 4bd42569fd3b4b39b0bd489acebbad10 P e47105a0f46e4511b7112e571688a4c7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:56:30.076545 16465 tablet_bootstrap.cc:492] T 4bd42569fd3b4b39b0bd489acebbad10 P e47105a0f46e4511b7112e571688a4c7: Bootstrap complete.
I20240503 13:56:30.076764 16465 ts_tablet_manager.cc:1367] T 4bd42569fd3b4b39b0bd489acebbad10 P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.033s user 0.022s sys 0.006s
I20240503 13:56:30.077047 16558 tablet_copy_client.cc:523] T 4647faf6a457415d8ae2b6691f452eab P 009f849ee7b54c80a3f5264695047e4d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:56:30.077226 16465 raft_consensus.cc:348] T 4bd42569fd3b4b39b0bd489acebbad10 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:30.077332 16465 raft_consensus.cc:729] T 4bd42569fd3b4b39b0bd489acebbad10 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:30.077512 16465 consensus_queue.cc:260] T 4bd42569fd3b4b39b0bd489acebbad10 P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:30.077875 16465 ts_tablet_manager.cc:1398] T 4bd42569fd3b4b39b0bd489acebbad10 P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:30.078403 13074 tablet_copy_service.cc:342] P 60ba3ffe3b7746469834cd91c84be14e: Request end of tablet copy session e47105a0f46e4511b7112e571688a4c7-4bd42569fd3b4b39b0bd489acebbad10 received from {username='slave'} at 127.6.98.72:59759
I20240503 13:56:30.078583 13074 tablet_copy_service.cc:433] P 60ba3ffe3b7746469834cd91c84be14e: ending tablet copy session e47105a0f46e4511b7112e571688a4c7-4bd42569fd3b4b39b0bd489acebbad10 on tablet 4bd42569fd3b4b39b0bd489acebbad10 with peer e47105a0f46e4511b7112e571688a4c7
I20240503 13:56:30.079630 16558 tablet_bootstrap.cc:492] T 4647faf6a457415d8ae2b6691f452eab P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:30.084367 13315 consensus_queue.cc:237] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:30.085917 13447 raft_consensus.cc:1261] T bb4f0bb47a29437090750c4486df2f02 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:56:30.085960 13183 raft_consensus.cc:1261] T bb4f0bb47a29437090750c4486df2f02 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 13:56:30.086216 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bb4f0bb47a29437090750c4486df2f02. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:30.086300 15154 consensus_queue.cc:1035] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:30.086443 16225 consensus_queue.cc:1035] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:30.088371 16225 raft_consensus.cc:2932] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:30.088800 13183 raft_consensus.cc:2932] T bb4f0bb47a29437090750c4486df2f02 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:30.091181 13447 raft_consensus.cc:2932] T bb4f0bb47a29437090750c4486df2f02 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:30.091730 12905 catalog_manager.cc:5484] T bb4f0bb47a29437090750c4486df2f02 P 8ccf5a4ac450414c9d3579bd5800b770 reported cstate change: config changed from index -1 to 4, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) added. New cstate: current_term: 1 leader_uuid: "ba0194098aa84449ae17e8a0d4c43703" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:30.095237 12908 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:56:30.100769 15889 raft_consensus.cc:1261] T 88137cb6ef4e4a6a96c5e8210fef0c96 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Refusing update from remote peer 76c386bef61a4bd5a907266a5118ffe2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:30.101163 15624 raft_consensus.cc:1261] T 88137cb6ef4e4a6a96c5e8210fef0c96 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Refusing update from remote peer 76c386bef61a4bd5a907266a5118ffe2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:30.101668 16266 consensus_queue.cc:1035] T 88137cb6ef4e4a6a96c5e8210fef0c96 P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:30.103826 16266 consensus_queue.cc:1035] T 88137cb6ef4e4a6a96c5e8210fef0c96 P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:30.104720 12909 catalog_manager.cc:5576] TS e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753) does not have the latest schema for tablet 88137cb6ef4e4a6a96c5e8210fef0c96 (table rebalancer_dynamic_table [id=0a1f605d9f474a4baf4693ac6e565608]). Expected version 1 got 0
I20240503 13:56:30.107017 12909 catalog_manager.cc:5576] TS 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951) does not have the latest schema for tablet 88137cb6ef4e4a6a96c5e8210fef0c96 (table rebalancer_dynamic_table [id=0a1f605d9f474a4baf4693ac6e565608]). Expected version 1 got 0
I20240503 13:56:30.108655 16484 ts_tablet_manager.cc:902] T b47b28a4766a425e8b9bdfea882f979d P 76c386bef61a4bd5a907266a5118ffe2: Initiating tablet copy from peer ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:30.108676 16186 tablet.cc:1722] T 88137cb6ef4e4a6a96c5e8210fef0c96 P 76c386bef61a4bd5a907266a5118ffe2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:30.109068 16186 tablet.cc:1620] T 88137cb6ef4e4a6a96c5e8210fef0c96 P 76c386bef61a4bd5a907266a5118ffe2: MemRowSet was empty: no flush needed.
I20240503 13:56:30.109082 16484 tablet_copy_client.cc:308] T b47b28a4766a425e8b9bdfea882f979d P 76c386bef61a4bd5a907266a5118ffe2: tablet copy: Beginning tablet copy session from remote peer at address 127.6.98.67:42279
I20240503 13:56:30.111744 16189 tablet.cc:1722] T 88137cb6ef4e4a6a96c5e8210fef0c96 P e47105a0f46e4511b7112e571688a4c7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:30.111972 16540 tablet_bootstrap.cc:492] T fd6a3d908840456f82c346c4bf1d8160 P 61b887f0d40d497badeca5604c1f0546: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:56:30.112079 16189 tablet.cc:1620] T 88137cb6ef4e4a6a96c5e8210fef0c96 P e47105a0f46e4511b7112e571688a4c7: MemRowSet was empty: no flush needed.
I20240503 13:56:30.112335 16540 tablet_bootstrap.cc:492] T fd6a3d908840456f82c346c4bf1d8160 P 61b887f0d40d497badeca5604c1f0546: Bootstrap complete.
I20240503 13:56:30.112617 16540 ts_tablet_manager.cc:1367] T fd6a3d908840456f82c346c4bf1d8160 P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.044s user 0.030s sys 0.001s
I20240503 13:56:30.113103 16540 raft_consensus.cc:348] T fd6a3d908840456f82c346c4bf1d8160 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:30.113305 16540 raft_consensus.cc:729] T fd6a3d908840456f82c346c4bf1d8160 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:30.113549 16540 consensus_queue.cc:260] T fd6a3d908840456f82c346c4bf1d8160 P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:30.115815 16540 ts_tablet_manager.cc:1398] T fd6a3d908840456f82c346c4bf1d8160 P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 13:56:30.117859 16182 mvcc.cc:205] Tried to move back new op lower bound from 7023593841047154688 to 7023593840532934656. Current Snapshot: MvccSnapshot[applied={T|T < 7023593841047154688}]
I20240503 13:56:30.118085 16187 tablet.cc:1722] T 88137cb6ef4e4a6a96c5e8210fef0c96 P 009f849ee7b54c80a3f5264695047e4d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:30.118387 16187 tablet.cc:1620] T 88137cb6ef4e4a6a96c5e8210fef0c96 P 009f849ee7b54c80a3f5264695047e4d: MemRowSet was empty: no flush needed.
I20240503 13:56:30.119081 13336 tablet_copy_service.cc:342] P ba0194098aa84449ae17e8a0d4c43703: Request end of tablet copy session 61b887f0d40d497badeca5604c1f0546-fd6a3d908840456f82c346c4bf1d8160 received from {username='slave'} at 127.6.98.69:36581
I20240503 13:56:30.119174 13336 tablet_copy_service.cc:433] P ba0194098aa84449ae17e8a0d4c43703: ending tablet copy session 61b887f0d40d497badeca5604c1f0546-fd6a3d908840456f82c346c4bf1d8160 on tablet fd6a3d908840456f82c346c4bf1d8160 with peer 61b887f0d40d497badeca5604c1f0546
I20240503 13:56:30.119856 12909 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=0a1f605d9f474a4baf4693ac6e565608] alter complete (version 1)
I20240503 13:56:30.125557 13336 tablet_copy_service.cc:140] P ba0194098aa84449ae17e8a0d4c43703: Received BeginTabletCopySession request for tablet b47b28a4766a425e8b9bdfea882f979d from peer 76c386bef61a4bd5a907266a5118ffe2 ({username='slave'} at 127.6.98.71:53369)
I20240503 13:56:30.125671 13336 tablet_copy_service.cc:161] P ba0194098aa84449ae17e8a0d4c43703: Beginning new tablet copy session on tablet b47b28a4766a425e8b9bdfea882f979d from peer 76c386bef61a4bd5a907266a5118ffe2 at {username='slave'} at 127.6.98.71:53369: session id = 76c386bef61a4bd5a907266a5118ffe2-b47b28a4766a425e8b9bdfea882f979d
I20240503 13:56:30.126502 13336 tablet_copy_source_session.cc:205] T b47b28a4766a425e8b9bdfea882f979d P ba0194098aa84449ae17e8a0d4c43703: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:56:30.127353 16484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b47b28a4766a425e8b9bdfea882f979d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:30.130177 16484 tablet_copy_client.cc:783] T b47b28a4766a425e8b9bdfea882f979d P 76c386bef61a4bd5a907266a5118ffe2: tablet copy: Starting download of 0 data blocks...
I20240503 13:56:30.131263 16484 tablet_copy_client.cc:647] T b47b28a4766a425e8b9bdfea882f979d P 76c386bef61a4bd5a907266a5118ffe2: tablet copy: Starting download of 1 WAL segments...
I20240503 13:56:30.131614 16540 ts_tablet_manager.cc:902] T b291e3dd6a7a454ca4d736eff8dd3a41 P 61b887f0d40d497badeca5604c1f0546: Initiating tablet copy from peer 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:30.132313 16540 tablet_copy_client.cc:308] T b291e3dd6a7a454ca4d736eff8dd3a41 P 61b887f0d40d497badeca5604c1f0546: tablet copy: Beginning tablet copy session from remote peer at address 127.6.98.68:36109
I20240503 13:56:30.133028 13467 tablet_copy_service.cc:140] P 7690ee79577c45e3a2c8f8b552c0bff8: Received BeginTabletCopySession request for tablet b291e3dd6a7a454ca4d736eff8dd3a41 from peer 61b887f0d40d497badeca5604c1f0546 ({username='slave'} at 127.6.98.69:40065)
I20240503 13:56:30.133138 13467 tablet_copy_service.cc:161] P 7690ee79577c45e3a2c8f8b552c0bff8: Beginning new tablet copy session on tablet b291e3dd6a7a454ca4d736eff8dd3a41 from peer 61b887f0d40d497badeca5604c1f0546 at {username='slave'} at 127.6.98.69:40065: session id = 61b887f0d40d497badeca5604c1f0546-b291e3dd6a7a454ca4d736eff8dd3a41
I20240503 13:56:30.133605 16484 tablet_copy_client.cc:523] T b47b28a4766a425e8b9bdfea882f979d P 76c386bef61a4bd5a907266a5118ffe2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:56:30.134191 13467 tablet_copy_source_session.cc:205] T b291e3dd6a7a454ca4d736eff8dd3a41 P 7690ee79577c45e3a2c8f8b552c0bff8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:56:30.135190 16484 tablet_bootstrap.cc:492] T b47b28a4766a425e8b9bdfea882f979d P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:30.136286 16540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b291e3dd6a7a454ca4d736eff8dd3a41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:30.138330 16540 tablet_copy_client.cc:783] T b291e3dd6a7a454ca4d736eff8dd3a41 P 61b887f0d40d497badeca5604c1f0546: tablet copy: Starting download of 0 data blocks...
I20240503 13:56:30.138621 16540 tablet_copy_client.cc:647] T b291e3dd6a7a454ca4d736eff8dd3a41 P 61b887f0d40d497badeca5604c1f0546: tablet copy: Starting download of 1 WAL segments...
I20240503 13:56:30.139835 16540 tablet_copy_client.cc:523] T b291e3dd6a7a454ca4d736eff8dd3a41 P 61b887f0d40d497badeca5604c1f0546: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:56:30.141875 16540 tablet_bootstrap.cc:492] T b291e3dd6a7a454ca4d736eff8dd3a41 P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:30.143383 16558 tablet_bootstrap.cc:492] T 4647faf6a457415d8ae2b6691f452eab P 009f849ee7b54c80a3f5264695047e4d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:56:30.143748 16558 tablet_bootstrap.cc:492] T 4647faf6a457415d8ae2b6691f452eab P 009f849ee7b54c80a3f5264695047e4d: Bootstrap complete.
I20240503 13:56:30.143956 16558 ts_tablet_manager.cc:1367] T 4647faf6a457415d8ae2b6691f452eab P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.064s user 0.028s sys 0.004s
I20240503 13:56:30.144435 16558 raft_consensus.cc:348] T 4647faf6a457415d8ae2b6691f452eab P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:30.144666 16558 raft_consensus.cc:729] T 4647faf6a457415d8ae2b6691f452eab P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:30.144928 16558 consensus_queue.cc:260] T 4647faf6a457415d8ae2b6691f452eab P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:30.145273 16558 ts_tablet_manager.cc:1398] T 4647faf6a457415d8ae2b6691f452eab P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:30.146133 13336 tablet_copy_service.cc:342] P ba0194098aa84449ae17e8a0d4c43703: Request end of tablet copy session 009f849ee7b54c80a3f5264695047e4d-4647faf6a457415d8ae2b6691f452eab received from {username='slave'} at 127.6.98.70:35625
I20240503 13:56:30.146219 13336 tablet_copy_service.cc:433] P ba0194098aa84449ae17e8a0d4c43703: ending tablet copy session 009f849ee7b54c80a3f5264695047e4d-4647faf6a457415d8ae2b6691f452eab on tablet 4647faf6a457415d8ae2b6691f452eab with peer 009f849ee7b54c80a3f5264695047e4d
I20240503 13:56:30.159129 16465 ts_tablet_manager.cc:902] T a746bcb2d718433f8c9acc8bd027a429 P e47105a0f46e4511b7112e571688a4c7: Initiating tablet copy from peer ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:30.159636 16465 tablet_copy_client.cc:308] T a746bcb2d718433f8c9acc8bd027a429 P e47105a0f46e4511b7112e571688a4c7: tablet copy: Beginning tablet copy session from remote peer at address 127.6.98.67:42279
I20240503 13:56:30.164348 13336 tablet_copy_service.cc:140] P ba0194098aa84449ae17e8a0d4c43703: Received BeginTabletCopySession request for tablet a746bcb2d718433f8c9acc8bd027a429 from peer e47105a0f46e4511b7112e571688a4c7 ({username='slave'} at 127.6.98.72:58879)
I20240503 13:56:30.164561 13336 tablet_copy_service.cc:161] P ba0194098aa84449ae17e8a0d4c43703: Beginning new tablet copy session on tablet a746bcb2d718433f8c9acc8bd027a429 from peer e47105a0f46e4511b7112e571688a4c7 at {username='slave'} at 127.6.98.72:58879: session id = e47105a0f46e4511b7112e571688a4c7-a746bcb2d718433f8c9acc8bd027a429
I20240503 13:56:30.165575 13336 tablet_copy_source_session.cc:205] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:56:30.165940 16484 tablet_bootstrap.cc:492] T b47b28a4766a425e8b9bdfea882f979d P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:56:30.166234 16484 tablet_bootstrap.cc:492] T b47b28a4766a425e8b9bdfea882f979d P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap complete.
I20240503 13:56:30.166481 16484 ts_tablet_manager.cc:1367] T b47b28a4766a425e8b9bdfea882f979d P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.031s user 0.025s sys 0.004s
I20240503 13:56:30.166885 16465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a746bcb2d718433f8c9acc8bd027a429. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:30.166972 16484 raft_consensus.cc:348] T b47b28a4766a425e8b9bdfea882f979d P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:30.167100 16484 raft_consensus.cc:729] T b47b28a4766a425e8b9bdfea882f979d P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:30.167263 16484 consensus_queue.cc:260] T b47b28a4766a425e8b9bdfea882f979d P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:30.168049 16484 ts_tablet_manager.cc:1398] T b47b28a4766a425e8b9bdfea882f979d P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:30.168543 13336 tablet_copy_service.cc:342] P ba0194098aa84449ae17e8a0d4c43703: Request end of tablet copy session 76c386bef61a4bd5a907266a5118ffe2-b47b28a4766a425e8b9bdfea882f979d received from {username='slave'} at 127.6.98.71:53369
I20240503 13:56:30.168617 13336 tablet_copy_service.cc:433] P ba0194098aa84449ae17e8a0d4c43703: ending tablet copy session 76c386bef61a4bd5a907266a5118ffe2-b47b28a4766a425e8b9bdfea882f979d on tablet b47b28a4766a425e8b9bdfea882f979d with peer 76c386bef61a4bd5a907266a5118ffe2
I20240503 13:56:30.168905 16465 tablet_copy_client.cc:783] T a746bcb2d718433f8c9acc8bd027a429 P e47105a0f46e4511b7112e571688a4c7: tablet copy: Starting download of 0 data blocks...
I20240503 13:56:30.169063 16465 tablet_copy_client.cc:647] T a746bcb2d718433f8c9acc8bd027a429 P e47105a0f46e4511b7112e571688a4c7: tablet copy: Starting download of 1 WAL segments...
I20240503 13:56:30.171880 16465 tablet_copy_client.cc:523] T a746bcb2d718433f8c9acc8bd027a429 P e47105a0f46e4511b7112e571688a4c7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:56:30.173170 16465 tablet_bootstrap.cc:492] T a746bcb2d718433f8c9acc8bd027a429 P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:30.174103 16540 tablet_bootstrap.cc:492] T b291e3dd6a7a454ca4d736eff8dd3a41 P 61b887f0d40d497badeca5604c1f0546: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:56:30.174321 16540 tablet_bootstrap.cc:492] T b291e3dd6a7a454ca4d736eff8dd3a41 P 61b887f0d40d497badeca5604c1f0546: Bootstrap complete.
I20240503 13:56:30.174455 16540 ts_tablet_manager.cc:1367] T b291e3dd6a7a454ca4d736eff8dd3a41 P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.033s user 0.023s sys 0.000s
I20240503 13:56:30.174890 16540 raft_consensus.cc:348] T b291e3dd6a7a454ca4d736eff8dd3a41 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:30.175004 16540 raft_consensus.cc:729] T b291e3dd6a7a454ca4d736eff8dd3a41 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:30.175160 16540 consensus_queue.cc:260] T b291e3dd6a7a454ca4d736eff8dd3a41 P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:30.176069 16540 ts_tablet_manager.cc:1398] T b291e3dd6a7a454ca4d736eff8dd3a41 P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 13:56:30.178530 13467 tablet_copy_service.cc:342] P 7690ee79577c45e3a2c8f8b552c0bff8: Request end of tablet copy session 61b887f0d40d497badeca5604c1f0546-b291e3dd6a7a454ca4d736eff8dd3a41 received from {username='slave'} at 127.6.98.69:40065
I20240503 13:56:30.178613 13467 tablet_copy_service.cc:433] P 7690ee79577c45e3a2c8f8b552c0bff8: ending tablet copy session 61b887f0d40d497badeca5604c1f0546-b291e3dd6a7a454ca4d736eff8dd3a41 on tablet b291e3dd6a7a454ca4d736eff8dd3a41 with peer 61b887f0d40d497badeca5604c1f0546
I20240503 13:56:30.179204 13052 consensus_queue.cc:237] T e9b5dc0762494e6ebdebb7b2833a272e P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:30.180801 13447 raft_consensus.cc:1261] T e9b5dc0762494e6ebdebb7b2833a272e P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer 60ba3ffe3b7746469834cd91c84be14e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 13:56:30.181015 12990 consensus_peers.cc:485] T e9b5dc0762494e6ebdebb7b2833a272e P 60ba3ffe3b7746469834cd91c84be14e -> Peer 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885): Couldn't send request to peer 76c386bef61a4bd5a907266a5118ffe2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e9b5dc0762494e6ebdebb7b2833a272e. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:30.181109 13315 raft_consensus.cc:1261] T e9b5dc0762494e6ebdebb7b2833a272e P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Refusing update from remote peer 60ba3ffe3b7746469834cd91c84be14e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:56:30.181502 15958 consensus_queue.cc:1035] T e9b5dc0762494e6ebdebb7b2833a272e P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:30.181965 15958 consensus_queue.cc:1035] T e9b5dc0762494e6ebdebb7b2833a272e P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:30.185788 16472 raft_consensus.cc:2932] T e9b5dc0762494e6ebdebb7b2833a272e P 60ba3ffe3b7746469834cd91c84be14e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:30.186144 13447 raft_consensus.cc:2932] T e9b5dc0762494e6ebdebb7b2833a272e P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:30.189116 12911 catalog_manager.cc:5484] T e9b5dc0762494e6ebdebb7b2833a272e P 60ba3ffe3b7746469834cd91c84be14e reported cstate change: config changed from index -1 to 4, VOTER 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71) added. New cstate: current_term: 1 leader_uuid: "60ba3ffe3b7746469834cd91c84be14e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:30.189882 13315 raft_consensus.cc:2932] T e9b5dc0762494e6ebdebb7b2833a272e P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:30.205475 16465 tablet_bootstrap.cc:492] T a746bcb2d718433f8c9acc8bd027a429 P e47105a0f46e4511b7112e571688a4c7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:56:30.205838 16465 tablet_bootstrap.cc:492] T a746bcb2d718433f8c9acc8bd027a429 P e47105a0f46e4511b7112e571688a4c7: Bootstrap complete.
I20240503 13:56:30.206053 16465 ts_tablet_manager.cc:1367] T a746bcb2d718433f8c9acc8bd027a429 P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.033s user 0.022s sys 0.002s
I20240503 13:56:30.206503 16465 raft_consensus.cc:348] T a746bcb2d718433f8c9acc8bd027a429 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:30.206612 16465 raft_consensus.cc:729] T a746bcb2d718433f8c9acc8bd027a429 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:30.206741 16465 consensus_queue.cc:260] T a746bcb2d718433f8c9acc8bd027a429 P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:30.207407 16465 ts_tablet_manager.cc:1398] T a746bcb2d718433f8c9acc8bd027a429 P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:30.207866 13336 tablet_copy_service.cc:342] P ba0194098aa84449ae17e8a0d4c43703: Request end of tablet copy session e47105a0f46e4511b7112e571688a4c7-a746bcb2d718433f8c9acc8bd027a429 received from {username='slave'} at 127.6.98.72:58879
I20240503 13:56:30.207970 13336 tablet_copy_service.cc:433] P ba0194098aa84449ae17e8a0d4c43703: ending tablet copy session e47105a0f46e4511b7112e571688a4c7-a746bcb2d718433f8c9acc8bd027a429 on tablet a746bcb2d718433f8c9acc8bd027a429 with peer e47105a0f46e4511b7112e571688a4c7
I20240503 13:56:30.220209 16540 ts_tablet_manager.cc:902] T 9eb0d911701b4b2bb33a6b7ab59f08aa P 61b887f0d40d497badeca5604c1f0546: Initiating tablet copy from peer ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:30.220688 16540 tablet_copy_client.cc:308] T 9eb0d911701b4b2bb33a6b7ab59f08aa P 61b887f0d40d497badeca5604c1f0546: tablet copy: Beginning tablet copy session from remote peer at address 127.6.98.67:42279
I20240503 13:56:30.221176 13336 tablet_copy_service.cc:140] P ba0194098aa84449ae17e8a0d4c43703: Received BeginTabletCopySession request for tablet 9eb0d911701b4b2bb33a6b7ab59f08aa from peer 61b887f0d40d497badeca5604c1f0546 ({username='slave'} at 127.6.98.69:36581)
I20240503 13:56:30.221302 13336 tablet_copy_service.cc:161] P ba0194098aa84449ae17e8a0d4c43703: Beginning new tablet copy session on tablet 9eb0d911701b4b2bb33a6b7ab59f08aa from peer 61b887f0d40d497badeca5604c1f0546 at {username='slave'} at 127.6.98.69:36581: session id = 61b887f0d40d497badeca5604c1f0546-9eb0d911701b4b2bb33a6b7ab59f08aa
I20240503 13:56:30.222250 13336 tablet_copy_source_session.cc:205] T 9eb0d911701b4b2bb33a6b7ab59f08aa P ba0194098aa84449ae17e8a0d4c43703: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:56:30.223389 12908 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:56:30.223721 16540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9eb0d911701b4b2bb33a6b7ab59f08aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:30.225840 16540 tablet_copy_client.cc:783] T 9eb0d911701b4b2bb33a6b7ab59f08aa P 61b887f0d40d497badeca5604c1f0546: tablet copy: Starting download of 0 data blocks...
I20240503 13:56:30.226004 16540 tablet_copy_client.cc:647] T 9eb0d911701b4b2bb33a6b7ab59f08aa P 61b887f0d40d497badeca5604c1f0546: tablet copy: Starting download of 1 WAL segments...
I20240503 13:56:30.228389 16540 tablet_copy_client.cc:523] T 9eb0d911701b4b2bb33a6b7ab59f08aa P 61b887f0d40d497badeca5604c1f0546: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:56:30.229739 16540 tablet_bootstrap.cc:492] T 9eb0d911701b4b2bb33a6b7ab59f08aa P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:30.229879 16186 tablet.cc:1722] T 88137cb6ef4e4a6a96c5e8210fef0c96 P 76c386bef61a4bd5a907266a5118ffe2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:30.230180 16186 tablet.cc:1620] T 88137cb6ef4e4a6a96c5e8210fef0c96 P 76c386bef61a4bd5a907266a5118ffe2: MemRowSet was empty: no flush needed.
I20240503 13:56:30.230237 16189 tablet.cc:1722] T 88137cb6ef4e4a6a96c5e8210fef0c96 P e47105a0f46e4511b7112e571688a4c7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:30.230552 16189 tablet.cc:1620] T 88137cb6ef4e4a6a96c5e8210fef0c96 P e47105a0f46e4511b7112e571688a4c7: MemRowSet was empty: no flush needed.
I20240503 13:56:30.232923 16187 tablet.cc:1722] T 88137cb6ef4e4a6a96c5e8210fef0c96 P 009f849ee7b54c80a3f5264695047e4d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:30.233240 16187 tablet.cc:1620] T 88137cb6ef4e4a6a96c5e8210fef0c96 P 009f849ee7b54c80a3f5264695047e4d: MemRowSet was empty: no flush needed.
I20240503 13:56:30.238729 12908 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=0a1f605d9f474a4baf4693ac6e565608] alter complete (version 2)
I20240503 13:56:30.240566 16345 consensus_queue.cc:1035] T a3c4cbf779554d8b84a48de78a9c392e P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:30.244057 16345 consensus_queue.cc:1035] T a3c4cbf779554d8b84a48de78a9c392e P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:30.263284 16540 tablet_bootstrap.cc:492] T 9eb0d911701b4b2bb33a6b7ab59f08aa P 61b887f0d40d497badeca5604c1f0546: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:56:30.263653 16540 tablet_bootstrap.cc:492] T 9eb0d911701b4b2bb33a6b7ab59f08aa P 61b887f0d40d497badeca5604c1f0546: Bootstrap complete.
I20240503 13:56:30.263906 16540 ts_tablet_manager.cc:1367] T 9eb0d911701b4b2bb33a6b7ab59f08aa P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.034s user 0.023s sys 0.001s
I20240503 13:56:30.264456 16540 raft_consensus.cc:348] T 9eb0d911701b4b2bb33a6b7ab59f08aa P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:30.264722 16540 raft_consensus.cc:729] T 9eb0d911701b4b2bb33a6b7ab59f08aa P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:30.264935 16540 consensus_queue.cc:260] T 9eb0d911701b4b2bb33a6b7ab59f08aa P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:30.265304 16540 ts_tablet_manager.cc:1398] T 9eb0d911701b4b2bb33a6b7ab59f08aa P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:30.266103 13336 tablet_copy_service.cc:342] P ba0194098aa84449ae17e8a0d4c43703: Request end of tablet copy session 61b887f0d40d497badeca5604c1f0546-9eb0d911701b4b2bb33a6b7ab59f08aa received from {username='slave'} at 127.6.98.69:36581
I20240503 13:56:30.266188 13336 tablet_copy_service.cc:433] P ba0194098aa84449ae17e8a0d4c43703: ending tablet copy session 61b887f0d40d497badeca5604c1f0546-9eb0d911701b4b2bb33a6b7ab59f08aa on tablet 9eb0d911701b4b2bb33a6b7ab59f08aa with peer 61b887f0d40d497badeca5604c1f0546
I20240503 13:56:30.276291 16558 ts_tablet_manager.cc:902] T f812304076e14c2db3dc02d1ff5766fb P 009f849ee7b54c80a3f5264695047e4d: Initiating tablet copy from peer 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:30.276841 16558 tablet_copy_client.cc:308] T f812304076e14c2db3dc02d1ff5766fb P 009f849ee7b54c80a3f5264695047e4d: tablet copy: Beginning tablet copy session from remote peer at address 127.6.98.68:36109
I20240503 13:56:30.281111 13467 tablet_copy_service.cc:140] P 7690ee79577c45e3a2c8f8b552c0bff8: Received BeginTabletCopySession request for tablet f812304076e14c2db3dc02d1ff5766fb from peer 009f849ee7b54c80a3f5264695047e4d ({username='slave'} at 127.6.98.70:40885)
I20240503 13:56:30.281216 13467 tablet_copy_service.cc:161] P 7690ee79577c45e3a2c8f8b552c0bff8: Beginning new tablet copy session on tablet f812304076e14c2db3dc02d1ff5766fb from peer 009f849ee7b54c80a3f5264695047e4d at {username='slave'} at 127.6.98.70:40885: session id = 009f849ee7b54c80a3f5264695047e4d-f812304076e14c2db3dc02d1ff5766fb
I20240503 13:56:30.282163 13467 tablet_copy_source_session.cc:205] T f812304076e14c2db3dc02d1ff5766fb P 7690ee79577c45e3a2c8f8b552c0bff8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:56:30.282989 16558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f812304076e14c2db3dc02d1ff5766fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:30.285944 16558 tablet_copy_client.cc:783] T f812304076e14c2db3dc02d1ff5766fb P 009f849ee7b54c80a3f5264695047e4d: tablet copy: Starting download of 0 data blocks...
I20240503 13:56:30.286104 16558 tablet_copy_client.cc:647] T f812304076e14c2db3dc02d1ff5766fb P 009f849ee7b54c80a3f5264695047e4d: tablet copy: Starting download of 1 WAL segments...
I20240503 13:56:30.287253 16558 tablet_copy_client.cc:523] T f812304076e14c2db3dc02d1ff5766fb P 009f849ee7b54c80a3f5264695047e4d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:56:30.288450 16558 tablet_bootstrap.cc:492] T f812304076e14c2db3dc02d1ff5766fb P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:30.296000 16404 consensus_queue.cc:1035] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:30.302459 16161 consensus_queue.cc:1035] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:30.319734 13034 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:30.319782 15604 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:56:30.322868 16558 tablet_bootstrap.cc:492] T f812304076e14c2db3dc02d1ff5766fb P 009f849ee7b54c80a3f5264695047e4d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:56:30.323216 16558 tablet_bootstrap.cc:492] T f812304076e14c2db3dc02d1ff5766fb P 009f849ee7b54c80a3f5264695047e4d: Bootstrap complete.
I20240503 13:56:30.323858 15472 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:56:30.324987 16558 ts_tablet_manager.cc:1367] T f812304076e14c2db3dc02d1ff5766fb P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.037s user 0.023s sys 0.003s
I20240503 13:56:30.329130 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:56:30.338171 16558 raft_consensus.cc:348] T f812304076e14c2db3dc02d1ff5766fb P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:30.338325 16558 raft_consensus.cc:729] T f812304076e14c2db3dc02d1ff5766fb P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:30.338472 16558 consensus_queue.cc:260] T f812304076e14c2db3dc02d1ff5766fb P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:30.338788 16558 ts_tablet_manager.cc:1398] T f812304076e14c2db3dc02d1ff5766fb P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.014s user 0.001s sys 0.000s
I20240503 13:56:30.341121 13467 tablet_copy_service.cc:342] P 7690ee79577c45e3a2c8f8b552c0bff8: Request end of tablet copy session 009f849ee7b54c80a3f5264695047e4d-f812304076e14c2db3dc02d1ff5766fb received from {username='slave'} at 127.6.98.70:40885
I20240503 13:56:30.341213 13467 tablet_copy_service.cc:433] P 7690ee79577c45e3a2c8f8b552c0bff8: ending tablet copy session 009f849ee7b54c80a3f5264695047e4d-f812304076e14c2db3dc02d1ff5766fb on tablet f812304076e14c2db3dc02d1ff5766fb with peer 009f849ee7b54c80a3f5264695047e4d
I20240503 13:56:30.342124 12904 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:56:30.348874 13296 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 13:56:30.350023 13164 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 13:56:30.350385 16484 ts_tablet_manager.cc:902] T 1b714fbd920a447f866b8d4547275675 P 76c386bef61a4bd5a907266a5118ffe2: Initiating tablet copy from peer ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:30.350895 16484 tablet_copy_client.cc:308] T 1b714fbd920a447f866b8d4547275675 P 76c386bef61a4bd5a907266a5118ffe2: tablet copy: Beginning tablet copy session from remote peer at address 127.6.98.67:42279
I20240503 13:56:30.352094 13336 tablet_copy_service.cc:140] P ba0194098aa84449ae17e8a0d4c43703: Received BeginTabletCopySession request for tablet 1b714fbd920a447f866b8d4547275675 from peer 76c386bef61a4bd5a907266a5118ffe2 ({username='slave'} at 127.6.98.71:53369)
I20240503 13:56:30.352195 13336 tablet_copy_service.cc:161] P ba0194098aa84449ae17e8a0d4c43703: Beginning new tablet copy session on tablet 1b714fbd920a447f866b8d4547275675 from peer 76c386bef61a4bd5a907266a5118ffe2 at {username='slave'} at 127.6.98.71:53369: session id = 76c386bef61a4bd5a907266a5118ffe2-1b714fbd920a447f866b8d4547275675
I20240503 13:56:30.353130 13336 tablet_copy_source_session.cc:205] T 1b714fbd920a447f866b8d4547275675 P ba0194098aa84449ae17e8a0d4c43703: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:56:30.357789 16186 tablet.cc:1722] T 88137cb6ef4e4a6a96c5e8210fef0c96 P 76c386bef61a4bd5a907266a5118ffe2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:30.359886 16189 tablet.cc:1722] T 88137cb6ef4e4a6a96c5e8210fef0c96 P e47105a0f46e4511b7112e571688a4c7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:30.359972 16187 tablet.cc:1722] T 88137cb6ef4e4a6a96c5e8210fef0c96 P 009f849ee7b54c80a3f5264695047e4d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:30.362179 16484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b714fbd920a447f866b8d4547275675. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:30.364311 16484 tablet_copy_client.cc:783] T 1b714fbd920a447f866b8d4547275675 P 76c386bef61a4bd5a907266a5118ffe2: tablet copy: Starting download of 0 data blocks...
I20240503 13:56:30.364464 16484 tablet_copy_client.cc:647] T 1b714fbd920a447f866b8d4547275675 P 76c386bef61a4bd5a907266a5118ffe2: tablet copy: Starting download of 1 WAL segments...
I20240503 13:56:30.365862 12911 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table_new_name [id=0a1f605d9f474a4baf4693ac6e565608] alter complete (version 3)
I20240503 13:56:30.366732 16484 tablet_copy_client.cc:523] T 1b714fbd920a447f866b8d4547275675 P 76c386bef61a4bd5a907266a5118ffe2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:56:30.367954 16484 tablet_bootstrap.cc:492] T 1b714fbd920a447f866b8d4547275675 P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:30.370857 15738 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:30.371562 15869 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:30.394169 15758 raft_consensus.cc:1203] T 81d47f705c0b4128a84e4bf7d274a659 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:56:30.400624 16484 tablet_bootstrap.cc:492] T 1b714fbd920a447f866b8d4547275675 P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:56:30.401082 16484 tablet_bootstrap.cc:492] T 1b714fbd920a447f866b8d4547275675 P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap complete.
I20240503 13:56:30.401265 16484 ts_tablet_manager.cc:1367] T 1b714fbd920a447f866b8d4547275675 P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.033s user 0.025s sys 0.000s
I20240503 13:56:30.401726 16484 raft_consensus.cc:348] T 1b714fbd920a447f866b8d4547275675 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:30.401928 16484 raft_consensus.cc:729] T 1b714fbd920a447f866b8d4547275675 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:30.402122 16484 consensus_queue.cc:260] T 1b714fbd920a447f866b8d4547275675 P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:30.402455 16484 ts_tablet_manager.cc:1398] T 1b714fbd920a447f866b8d4547275675 P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 13:56:30.404420 13336 tablet_copy_service.cc:342] P ba0194098aa84449ae17e8a0d4c43703: Request end of tablet copy session 76c386bef61a4bd5a907266a5118ffe2-1b714fbd920a447f866b8d4547275675 received from {username='slave'} at 127.6.98.71:53369
I20240503 13:56:30.404537 13336 tablet_copy_service.cc:433] P ba0194098aa84449ae17e8a0d4c43703: ending tablet copy session 76c386bef61a4bd5a907266a5118ffe2-1b714fbd920a447f866b8d4547275675 on tablet 1b714fbd920a447f866b8d4547275675 with peer 76c386bef61a4bd5a907266a5118ffe2
I20240503 13:56:30.413745 12904 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalance_test_table_6" } modify_external_catalogs: true
I20240503 13:56:30.413915 12904 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalance_test_table_6" } modify_external_catalogs: true
I20240503 13:56:30.416684 12904 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Sending DeleteTablet for 3 replicas of tablet 11f710cf381f4166b9625a6bb8646a5f
I20240503 13:56:30.417243 13427 tablet_service.cc:1514] Processing DeleteTablet for tablet 11f710cf381f4166b9625a6bb8646a5f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:30 UTC) from {username='slave'} at 127.0.0.1:33678
I20240503 13:56:30.417284 12904 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Sending DeleteTablet for 4 replicas of tablet b923fd3f226e409b813bb4e63325a7fc
I20240503 13:56:30.417593 16087 tablet_replica.cc:330] T 11f710cf381f4166b9625a6bb8646a5f P 7690ee79577c45e3a2c8f8b552c0bff8: stopping tablet replica
I20240503 13:56:30.417575 13034 tablet_service.cc:1514] Processing DeleteTablet for tablet 11f710cf381f4166b9625a6bb8646a5f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:30 UTC) from {username='slave'} at 127.0.0.1:59458
I20240503 13:56:30.417582 13164 tablet_service.cc:1514] Processing DeleteTablet for tablet 11f710cf381f4166b9625a6bb8646a5f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:30 UTC) from {username='slave'} at 127.0.0.1:41668
I20240503 13:56:30.417729 16087 raft_consensus.cc:2229] T 11f710cf381f4166b9625a6bb8646a5f P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:56:30.417752 16085 tablet_replica.cc:330] T 11f710cf381f4166b9625a6bb8646a5f P 60ba3ffe3b7746469834cd91c84be14e: stopping tablet replica
I20240503 13:56:30.417755 16092 tablet_replica.cc:330] T 11f710cf381f4166b9625a6bb8646a5f P 8ccf5a4ac450414c9d3579bd5800b770: stopping tablet replica
I20240503 13:56:30.417863 16085 raft_consensus.cc:2229] T 11f710cf381f4166b9625a6bb8646a5f P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:30.417876 16092 raft_consensus.cc:2229] T 11f710cf381f4166b9625a6bb8646a5f P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:30.417920 16087 raft_consensus.cc:2258] T 11f710cf381f4166b9625a6bb8646a5f P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:30.417958 16085 raft_consensus.cc:2258] T 11f710cf381f4166b9625a6bb8646a5f P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:30.417992 16092 raft_consensus.cc:2258] T 11f710cf381f4166b9625a6bb8646a5f P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:30.418365 16085 ts_tablet_manager.cc:1875] T 11f710cf381f4166b9625a6bb8646a5f P 60ba3ffe3b7746469834cd91c84be14e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:30.418423 16092 ts_tablet_manager.cc:1875] T 11f710cf381f4166b9625a6bb8646a5f P 8ccf5a4ac450414c9d3579bd5800b770: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:30.418411 16087 ts_tablet_manager.cc:1875] T 11f710cf381f4166b9625a6bb8646a5f P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:30.420554 16085 ts_tablet_manager.cc:1888] T 11f710cf381f4166b9625a6bb8646a5f P 60ba3ffe3b7746469834cd91c84be14e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:56:30.420593 16087 ts_tablet_manager.cc:1888] T 11f710cf381f4166b9625a6bb8646a5f P 7690ee79577c45e3a2c8f8b552c0bff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:56:30.420593 16092 ts_tablet_manager.cc:1888] T 11f710cf381f4166b9625a6bb8646a5f P 8ccf5a4ac450414c9d3579bd5800b770: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:56:30.420626 16085 log.cc:1196] T 11f710cf381f4166b9625a6bb8646a5f P 60ba3ffe3b7746469834cd91c84be14e: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal/wals/11f710cf381f4166b9625a6bb8646a5f
I20240503 13:56:30.420655 16092 log.cc:1196] T 11f710cf381f4166b9625a6bb8646a5f P 8ccf5a4ac450414c9d3579bd5800b770: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/wal/wals/11f710cf381f4166b9625a6bb8646a5f
I20240503 13:56:30.420655 16087 log.cc:1196] T 11f710cf381f4166b9625a6bb8646a5f P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal/wals/11f710cf381f4166b9625a6bb8646a5f
I20240503 13:56:30.420847 16085 ts_tablet_manager.cc:1909] T 11f710cf381f4166b9625a6bb8646a5f P 60ba3ffe3b7746469834cd91c84be14e: Deleting consensus metadata
I20240503 13:56:30.420908 16087 ts_tablet_manager.cc:1909] T 11f710cf381f4166b9625a6bb8646a5f P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting consensus metadata
I20240503 13:56:30.420908 16092 ts_tablet_manager.cc:1909] T 11f710cf381f4166b9625a6bb8646a5f P 8ccf5a4ac450414c9d3579bd5800b770: Deleting consensus metadata
I20240503 13:56:30.421381 13034 tablet_service.cc:1514] Processing DeleteTablet for tablet b923fd3f226e409b813bb4e63325a7fc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:30 UTC) from {username='slave'} at 127.0.0.1:59458
I20240503 13:56:30.421576 12897 catalog_manager.cc:4830] TS 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695): tablet 11f710cf381f4166b9625a6bb8646a5f (table rebalance_test_table_6 [id=89b1c278366c4030ac7c6b90c882c67e]) successfully deleted
I20240503 13:56:30.421799 13296 tablet_service.cc:1514] Processing DeleteTablet for tablet b923fd3f226e409b813bb4e63325a7fc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:30 UTC) from {username='slave'} at 127.0.0.1:33250
I20240503 13:56:30.421852 12896 catalog_manager.cc:4830] TS 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109): tablet 11f710cf381f4166b9625a6bb8646a5f (table rebalance_test_table_6 [id=89b1c278366c4030ac7c6b90c882c67e]) successfully deleted
I20240503 13:56:30.422019 16086 tablet_replica.cc:330] T b923fd3f226e409b813bb4e63325a7fc P ba0194098aa84449ae17e8a0d4c43703: stopping tablet replica
I20240503 13:56:30.422138 16086 raft_consensus.cc:2229] T b923fd3f226e409b813bb4e63325a7fc P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:30.422334 16086 raft_consensus.cc:2258] T b923fd3f226e409b813bb4e63325a7fc P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:30.422336 12899 catalog_manager.cc:4830] TS 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387): tablet 11f710cf381f4166b9625a6bb8646a5f (table rebalance_test_table_6 [id=89b1c278366c4030ac7c6b90c882c67e]) successfully deleted
I20240503 13:56:30.422508 16085 tablet_replica.cc:330] T b923fd3f226e409b813bb4e63325a7fc P 60ba3ffe3b7746469834cd91c84be14e: stopping tablet replica
I20240503 13:56:30.422693 16085 raft_consensus.cc:2229] T b923fd3f226e409b813bb4e63325a7fc P 60ba3ffe3b7746469834cd91c84be14e [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:56:30.422986 16085 raft_consensus.cc:2258] T b923fd3f226e409b813bb4e63325a7fc P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:30.423035 16086 ts_tablet_manager.cc:1875] T b923fd3f226e409b813bb4e63325a7fc P ba0194098aa84449ae17e8a0d4c43703: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:30.423662 12904 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Sending DeleteTablet for 4 replicas of tablet f812304076e14c2db3dc02d1ff5766fb
I20240503 13:56:30.423962 13427 tablet_service.cc:1514] Processing DeleteTablet for tablet b923fd3f226e409b813bb4e63325a7fc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:30 UTC) from {username='slave'} at 127.0.0.1:33678
I20240503 13:56:30.424119 13034 tablet_service.cc:1514] Processing DeleteTablet for tablet f812304076e14c2db3dc02d1ff5766fb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:30 UTC) from {username='slave'} at 127.0.0.1:59458
I20240503 13:56:30.424264 16087 tablet_replica.cc:330] T b923fd3f226e409b813bb4e63325a7fc P 7690ee79577c45e3a2c8f8b552c0bff8: stopping tablet replica
I20240503 13:56:30.424283 16085 ts_tablet_manager.cc:1875] T b923fd3f226e409b813bb4e63325a7fc P 60ba3ffe3b7746469834cd91c84be14e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:30.424396 16087 raft_consensus.cc:2229] T b923fd3f226e409b813bb4e63325a7fc P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:30.424608 16087 raft_consensus.cc:2258] T b923fd3f226e409b813bb4e63325a7fc P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:30.424793 15869 tablet_service.cc:1514] Processing DeleteTablet for tablet b923fd3f226e409b813bb4e63325a7fc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:30 UTC) from {username='slave'} at 127.0.0.1:50498
I20240503 13:56:30.424846 13164 tablet_service.cc:1514] Processing DeleteTablet for tablet f812304076e14c2db3dc02d1ff5766fb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:30 UTC) from {username='slave'} at 127.0.0.1:41668
I20240503 13:56:30.425107 16684 tablet_replica.cc:330] T b923fd3f226e409b813bb4e63325a7fc P e47105a0f46e4511b7112e571688a4c7: stopping tablet replica
I20240503 13:56:30.425323 13427 tablet_service.cc:1514] Processing DeleteTablet for tablet f812304076e14c2db3dc02d1ff5766fb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:30 UTC) from {username='slave'} at 127.0.0.1:33678
I20240503 13:56:30.425311 16684 raft_consensus.cc:2229] T b923fd3f226e409b813bb4e63325a7fc P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:30.425438 16684 raft_consensus.cc:2258] T b923fd3f226e409b813bb4e63325a7fc P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:30.425721 16086 ts_tablet_manager.cc:1888] T b923fd3f226e409b813bb4e63325a7fc P ba0194098aa84449ae17e8a0d4c43703: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:30.425796 16086 log.cc:1196] T b923fd3f226e409b813bb4e63325a7fc P ba0194098aa84449ae17e8a0d4c43703: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/wal/wals/b923fd3f226e409b813bb4e63325a7fc
I20240503 13:56:30.425921 16087 ts_tablet_manager.cc:1875] T b923fd3f226e409b813bb4e63325a7fc P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:30.426048 16086 ts_tablet_manager.cc:1909] T b923fd3f226e409b813bb4e63325a7fc P ba0194098aa84449ae17e8a0d4c43703: Deleting consensus metadata
I20240503 13:56:30.426383 15604 tablet_service.cc:1514] Processing DeleteTablet for tablet f812304076e14c2db3dc02d1ff5766fb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:30 UTC) from {username='slave'} at 127.0.0.1:60718
I20240503 13:56:30.426791 12899 catalog_manager.cc:4830] TS ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279): tablet b923fd3f226e409b813bb4e63325a7fc (table rebalance_test_table_6 [id=89b1c278366c4030ac7c6b90c882c67e]) successfully deleted
I20240503 13:56:30.426811 16092 tablet_replica.cc:330] T f812304076e14c2db3dc02d1ff5766fb P 8ccf5a4ac450414c9d3579bd5800b770: stopping tablet replica
I20240503 13:56:30.427021 16092 raft_consensus.cc:2229] T f812304076e14c2db3dc02d1ff5766fb P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:30.427026 16684 ts_tablet_manager.cc:1875] T b923fd3f226e409b813bb4e63325a7fc P e47105a0f46e4511b7112e571688a4c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:30.427212 16092 raft_consensus.cc:2258] T f812304076e14c2db3dc02d1ff5766fb P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:30.427826 16092 ts_tablet_manager.cc:1875] T f812304076e14c2db3dc02d1ff5766fb P 8ccf5a4ac450414c9d3579bd5800b770: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:30.427932 16087 ts_tablet_manager.cc:1888] T b923fd3f226e409b813bb4e63325a7fc P 7690ee79577c45e3a2c8f8b552c0bff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:30.427999 16087 log.cc:1196] T b923fd3f226e409b813bb4e63325a7fc P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal/wals/b923fd3f226e409b813bb4e63325a7fc
I20240503 13:56:30.428238 16087 ts_tablet_manager.cc:1909] T b923fd3f226e409b813bb4e63325a7fc P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting consensus metadata
I20240503 13:56:30.429425 16684 ts_tablet_manager.cc:1888] T b923fd3f226e409b813bb4e63325a7fc P e47105a0f46e4511b7112e571688a4c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:30.429538 16684 log.cc:1196] T b923fd3f226e409b813bb4e63325a7fc P e47105a0f46e4511b7112e571688a4c7: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-7/wal/wals/b923fd3f226e409b813bb4e63325a7fc
I20240503 13:56:30.429795 16684 ts_tablet_manager.cc:1909] T b923fd3f226e409b813bb4e63325a7fc P e47105a0f46e4511b7112e571688a4c7: Deleting consensus metadata
I20240503 13:56:30.430162 16092 ts_tablet_manager.cc:1888] T f812304076e14c2db3dc02d1ff5766fb P 8ccf5a4ac450414c9d3579bd5800b770: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:30.430241 16092 log.cc:1196] T f812304076e14c2db3dc02d1ff5766fb P 8ccf5a4ac450414c9d3579bd5800b770: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/wal/wals/f812304076e14c2db3dc02d1ff5766fb
I20240503 13:56:30.430362 16085 ts_tablet_manager.cc:1888] T b923fd3f226e409b813bb4e63325a7fc P 60ba3ffe3b7746469834cd91c84be14e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:30.430433 16085 log.cc:1196] T b923fd3f226e409b813bb4e63325a7fc P 60ba3ffe3b7746469834cd91c84be14e: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal/wals/b923fd3f226e409b813bb4e63325a7fc
I20240503 13:56:30.430588 16092 ts_tablet_manager.cc:1909] T f812304076e14c2db3dc02d1ff5766fb P 8ccf5a4ac450414c9d3579bd5800b770: Deleting consensus metadata
I20240503 13:56:30.430617 12896 catalog_manager.cc:4830] TS e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): tablet b923fd3f226e409b813bb4e63325a7fc (table rebalance_test_table_6 [id=89b1c278366c4030ac7c6b90c882c67e]) successfully deleted
I20240503 13:56:30.430681 16085 ts_tablet_manager.cc:1909] T b923fd3f226e409b813bb4e63325a7fc P 60ba3ffe3b7746469834cd91c84be14e: Deleting consensus metadata
I20240503 13:56:30.431232 16087 tablet_replica.cc:330] T f812304076e14c2db3dc02d1ff5766fb P 7690ee79577c45e3a2c8f8b552c0bff8: stopping tablet replica
I20240503 13:56:30.431299 12899 catalog_manager.cc:4830] TS 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387): tablet f812304076e14c2db3dc02d1ff5766fb (table rebalance_test_table_6 [id=89b1c278366c4030ac7c6b90c882c67e]) successfully deleted
I20240503 13:56:30.431339 12896 catalog_manager.cc:4830] TS 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109): tablet b923fd3f226e409b813bb4e63325a7fc (table rebalance_test_table_6 [id=89b1c278366c4030ac7c6b90c882c67e]) successfully deleted
I20240503 13:56:30.431362 16087 raft_consensus.cc:2229] T f812304076e14c2db3dc02d1ff5766fb P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:56:30.431365 12897 catalog_manager.cc:4830] TS 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695): tablet b923fd3f226e409b813bb4e63325a7fc (table rebalance_test_table_6 [id=89b1c278366c4030ac7c6b90c882c67e]) successfully deleted
I20240503 13:56:30.431581 16085 tablet_replica.cc:330] T f812304076e14c2db3dc02d1ff5766fb P 60ba3ffe3b7746469834cd91c84be14e: stopping tablet replica
I20240503 13:56:30.431620 16087 raft_consensus.cc:2258] T f812304076e14c2db3dc02d1ff5766fb P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:30.431711 16085 raft_consensus.cc:2229] T f812304076e14c2db3dc02d1ff5766fb P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:30.431874 16085 raft_consensus.cc:2258] T f812304076e14c2db3dc02d1ff5766fb P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:30.432387 16085 ts_tablet_manager.cc:1875] T f812304076e14c2db3dc02d1ff5766fb P 60ba3ffe3b7746469834cd91c84be14e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:30.432847 16087 ts_tablet_manager.cc:1875] T f812304076e14c2db3dc02d1ff5766fb P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:30.434234 16085 ts_tablet_manager.cc:1888] T f812304076e14c2db3dc02d1ff5766fb P 60ba3ffe3b7746469834cd91c84be14e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:30.434306 16085 log.cc:1196] T f812304076e14c2db3dc02d1ff5766fb P 60ba3ffe3b7746469834cd91c84be14e: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal/wals/f812304076e14c2db3dc02d1ff5766fb
I20240503 13:56:30.434540 16085 ts_tablet_manager.cc:1909] T f812304076e14c2db3dc02d1ff5766fb P 60ba3ffe3b7746469834cd91c84be14e: Deleting consensus metadata
I20240503 13:56:30.435241 12897 catalog_manager.cc:4830] TS 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695): tablet f812304076e14c2db3dc02d1ff5766fb (table rebalance_test_table_6 [id=89b1c278366c4030ac7c6b90c882c67e]) successfully deleted
I20240503 13:56:30.437016 16087 ts_tablet_manager.cc:1888] T f812304076e14c2db3dc02d1ff5766fb P 7690ee79577c45e3a2c8f8b552c0bff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:30.437086 16087 log.cc:1196] T f812304076e14c2db3dc02d1ff5766fb P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal/wals/f812304076e14c2db3dc02d1ff5766fb
I20240503 13:56:30.437312 16087 ts_tablet_manager.cc:1909] T f812304076e14c2db3dc02d1ff5766fb P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting consensus metadata
I20240503 13:56:30.437574 16685 tablet_replica.cc:330] T f812304076e14c2db3dc02d1ff5766fb P 009f849ee7b54c80a3f5264695047e4d: stopping tablet replica
I20240503 13:56:30.437737 16685 raft_consensus.cc:2229] T f812304076e14c2db3dc02d1ff5766fb P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:30.437897 16685 raft_consensus.cc:2258] T f812304076e14c2db3dc02d1ff5766fb P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:30.438282 12896 catalog_manager.cc:4830] TS 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109): tablet f812304076e14c2db3dc02d1ff5766fb (table rebalance_test_table_6 [id=89b1c278366c4030ac7c6b90c882c67e]) successfully deleted
I20240503 13:56:30.438552 16685 ts_tablet_manager.cc:1875] T f812304076e14c2db3dc02d1ff5766fb P 009f849ee7b54c80a3f5264695047e4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:30.440393 16685 ts_tablet_manager.cc:1888] T f812304076e14c2db3dc02d1ff5766fb P 009f849ee7b54c80a3f5264695047e4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:30.440454 16685 log.cc:1196] T f812304076e14c2db3dc02d1ff5766fb P 009f849ee7b54c80a3f5264695047e4d: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/wal/wals/f812304076e14c2db3dc02d1ff5766fb
I20240503 13:56:30.440694 16685 ts_tablet_manager.cc:1909] T f812304076e14c2db3dc02d1ff5766fb P 009f849ee7b54c80a3f5264695047e4d: Deleting consensus metadata
I20240503 13:56:30.441466 12896 catalog_manager.cc:4830] TS 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951): tablet f812304076e14c2db3dc02d1ff5766fb (table rebalance_test_table_6 [id=89b1c278366c4030ac7c6b90c882c67e]) successfully deleted
I20240503 13:56:30.446285 15889 raft_consensus.cc:1203] T 4bd42569fd3b4b39b0bd489acebbad10 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:56:30.465940 16161 consensus_queue.cc:1035] T 0f9224c2e3b946d993a5fa5c95a1d681 P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:30.468677 13183 consensus_queue.cc:237] T 08e3096cdac34940a3ab0529cb76c3c7 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:30.469027 16404 consensus_queue.cc:1035] T 0f9224c2e3b946d993a5fa5c95a1d681 P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:30.470749 13446 raft_consensus.cc:1261] T 08e3096cdac34940a3ab0529cb76c3c7 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer 8ccf5a4ac450414c9d3579bd5800b770: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 13:56:30.470804 13118 consensus_peers.cc:485] T 08e3096cdac34940a3ab0529cb76c3c7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951): Couldn't send request to peer 009f849ee7b54c80a3f5264695047e4d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 08e3096cdac34940a3ab0529cb76c3c7. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:30.470963 13051 raft_consensus.cc:1261] T 08e3096cdac34940a3ab0529cb76c3c7 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Refusing update from remote peer 8ccf5a4ac450414c9d3579bd5800b770: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:56:30.471374 16345 consensus_queue.cc:1035] T 08e3096cdac34940a3ab0529cb76c3c7 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:30.472107 12904 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:30.472239 12904 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:30.474093 16704 consensus_queue.cc:1035] T 08e3096cdac34940a3ab0529cb76c3c7 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:30.474525 12904 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Sending DeleteTablet for 3 replicas of tablet 88137cb6ef4e4a6a96c5e8210fef0c96
I20240503 13:56:30.474809 16558 ts_tablet_manager.cc:902] T 866a258ecb2e4685908d5382cb594959 P 009f849ee7b54c80a3f5264695047e4d: Initiating tablet copy from peer 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:30.475116 15604 tablet_service.cc:1514] Processing DeleteTablet for tablet 88137cb6ef4e4a6a96c5e8210fef0c96 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:30 UTC) from {username='slave'} at 127.0.0.1:60718
I20240503 13:56:30.475276 16685 tablet_replica.cc:330] T 88137cb6ef4e4a6a96c5e8210fef0c96 P 009f849ee7b54c80a3f5264695047e4d: stopping tablet replica
I20240503 13:56:30.475325 16558 tablet_copy_client.cc:308] T 866a258ecb2e4685908d5382cb594959 P 009f849ee7b54c80a3f5264695047e4d: tablet copy: Beginning tablet copy session from remote peer at address 127.6.98.68:36109
I20240503 13:56:30.475387 16685 raft_consensus.cc:2229] T 88137cb6ef4e4a6a96c5e8210fef0c96 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:30.475488 16685 raft_consensus.cc:2258] T 88137cb6ef4e4a6a96c5e8210fef0c96 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:30.475601 15869 tablet_service.cc:1514] Processing DeleteTablet for tablet 88137cb6ef4e4a6a96c5e8210fef0c96 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:30 UTC) from {username='slave'} at 127.0.0.1:50498
I20240503 13:56:30.475772 16684 tablet_replica.cc:330] T 88137cb6ef4e4a6a96c5e8210fef0c96 P e47105a0f46e4511b7112e571688a4c7: stopping tablet replica
I20240503 13:56:30.475875 16684 raft_consensus.cc:2229] T 88137cb6ef4e4a6a96c5e8210fef0c96 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:30.475982 16684 raft_consensus.cc:2258] T 88137cb6ef4e4a6a96c5e8210fef0c96 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:30.476110 13467 tablet_copy_service.cc:140] P 7690ee79577c45e3a2c8f8b552c0bff8: Received BeginTabletCopySession request for tablet 866a258ecb2e4685908d5382cb594959 from peer 009f849ee7b54c80a3f5264695047e4d ({username='slave'} at 127.6.98.70:40885)
I20240503 13:56:30.476190 13467 tablet_copy_service.cc:161] P 7690ee79577c45e3a2c8f8b552c0bff8: Beginning new tablet copy session on tablet 866a258ecb2e4685908d5382cb594959 from peer 009f849ee7b54c80a3f5264695047e4d at {username='slave'} at 127.6.98.70:40885: session id = 009f849ee7b54c80a3f5264695047e4d-866a258ecb2e4685908d5382cb594959
I20240503 13:56:30.476303 16685 ts_tablet_manager.cc:1875] T 88137cb6ef4e4a6a96c5e8210fef0c96 P 009f849ee7b54c80a3f5264695047e4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:30.476605 16684 ts_tablet_manager.cc:1875] T 88137cb6ef4e4a6a96c5e8210fef0c96 P e47105a0f46e4511b7112e571688a4c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:30.476837 15738 tablet_service.cc:1514] Processing DeleteTablet for tablet 88137cb6ef4e4a6a96c5e8210fef0c96 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:30 UTC) from {username='slave'} at 127.0.0.1:44984
I20240503 13:56:30.477061 13467 tablet_copy_source_session.cc:205] T 866a258ecb2e4685908d5382cb594959 P 7690ee79577c45e3a2c8f8b552c0bff8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:56:30.477872 16558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 866a258ecb2e4685908d5382cb594959. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:30.478176 13446 raft_consensus.cc:2932] T 08e3096cdac34940a3ab0529cb76c3c7 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 009f849ee7b54c80a3f5264695047e4d (127.6.98.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:30.478456 16685 ts_tablet_manager.cc:1888] T 88137cb6ef4e4a6a96c5e8210fef0c96 P 009f849ee7b54c80a3f5264695047e4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:30.478467 16684 ts_tablet_manager.cc:1888] T 88137cb6ef4e4a6a96c5e8210fef0c96 P e47105a0f46e4511b7112e571688a4c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:30.478524 16685 log.cc:1196] T 88137cb6ef4e4a6a96c5e8210fef0c96 P 009f849ee7b54c80a3f5264695047e4d: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/wal/wals/88137cb6ef4e4a6a96c5e8210fef0c96
I20240503 13:56:30.478523 16684 log.cc:1196] T 88137cb6ef4e4a6a96c5e8210fef0c96 P e47105a0f46e4511b7112e571688a4c7: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-7/wal/wals/88137cb6ef4e4a6a96c5e8210fef0c96
I20240503 13:56:30.478767 16684 ts_tablet_manager.cc:1909] T 88137cb6ef4e4a6a96c5e8210fef0c96 P e47105a0f46e4511b7112e571688a4c7: Deleting consensus metadata
I20240503 13:56:30.478768 16685 ts_tablet_manager.cc:1909] T 88137cb6ef4e4a6a96c5e8210fef0c96 P 009f849ee7b54c80a3f5264695047e4d: Deleting consensus metadata
I20240503 13:56:30.479498 12896 catalog_manager.cc:4830] TS e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): tablet 88137cb6ef4e4a6a96c5e8210fef0c96 (table rebalancer_dynamic_table_new_name [id=0a1f605d9f474a4baf4693ac6e565608]) successfully deleted
I20240503 13:56:30.479660 12896 catalog_manager.cc:4830] TS 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951): tablet 88137cb6ef4e4a6a96c5e8210fef0c96 (table rebalancer_dynamic_table_new_name [id=0a1f605d9f474a4baf4693ac6e565608]) successfully deleted
I20240503 13:56:30.479800 16558 tablet_copy_client.cc:783] T 866a258ecb2e4685908d5382cb594959 P 009f849ee7b54c80a3f5264695047e4d: tablet copy: Starting download of 0 data blocks...
I20240503 13:56:30.479653 16091 raft_consensus.cc:2932] T 08e3096cdac34940a3ab0529cb76c3c7 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 009f849ee7b54c80a3f5264695047e4d (127.6.98.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:30.479933 16558 tablet_copy_client.cc:647] T 866a258ecb2e4685908d5382cb594959 P 009f849ee7b54c80a3f5264695047e4d: tablet copy: Starting download of 1 WAL segments...
I20240503 13:56:30.481173 16558 tablet_copy_client.cc:523] T 866a258ecb2e4685908d5382cb594959 P 009f849ee7b54c80a3f5264695047e4d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:56:30.481459 16712 tablet_replica.cc:330] T 88137cb6ef4e4a6a96c5e8210fef0c96 P 76c386bef61a4bd5a907266a5118ffe2: stopping tablet replica
I20240503 13:56:30.481596 16712 raft_consensus.cc:2229] T 88137cb6ef4e4a6a96c5e8210fef0c96 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:56:30.481794 16712 raft_consensus.cc:2258] T 88137cb6ef4e4a6a96c5e8210fef0c96 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:30.481734 12904 catalog_manager.cc:5484] T 08e3096cdac34940a3ab0529cb76c3c7 P 7690ee79577c45e3a2c8f8b552c0bff8 reported cstate change: config changed from index -1 to 4, VOTER 009f849ee7b54c80a3f5264695047e4d (127.6.98.70) added. New cstate: current_term: 1 leader_uuid: "8ccf5a4ac450414c9d3579bd5800b770" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:30.483044 13051 raft_consensus.cc:2932] T 08e3096cdac34940a3ab0529cb76c3c7 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 009f849ee7b54c80a3f5264695047e4d (127.6.98.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:30.484076 16558 tablet_bootstrap.cc:492] T 866a258ecb2e4685908d5382cb594959 P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:30.486724 16712 ts_tablet_manager.cc:1875] T 88137cb6ef4e4a6a96c5e8210fef0c96 P 76c386bef61a4bd5a907266a5118ffe2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:30.489271 16484 ts_tablet_manager.cc:902] T 05917bf47aea42d397ccae57c3f29416 P 76c386bef61a4bd5a907266a5118ffe2: Initiating tablet copy from peer 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:30.489742 16484 tablet_copy_client.cc:308] T 05917bf47aea42d397ccae57c3f29416 P 76c386bef61a4bd5a907266a5118ffe2: tablet copy: Beginning tablet copy session from remote peer at address 127.6.98.65:34695
I20240503 13:56:30.490482 13074 tablet_copy_service.cc:140] P 60ba3ffe3b7746469834cd91c84be14e: Received BeginTabletCopySession request for tablet 05917bf47aea42d397ccae57c3f29416 from peer 76c386bef61a4bd5a907266a5118ffe2 ({username='slave'} at 127.6.98.71:42057)
I20240503 13:56:30.490593 13074 tablet_copy_service.cc:161] P 60ba3ffe3b7746469834cd91c84be14e: Beginning new tablet copy session on tablet 05917bf47aea42d397ccae57c3f29416 from peer 76c386bef61a4bd5a907266a5118ffe2 at {username='slave'} at 127.6.98.71:42057: session id = 76c386bef61a4bd5a907266a5118ffe2-05917bf47aea42d397ccae57c3f29416
I20240503 13:56:30.491528 13074 tablet_copy_source_session.cc:205] T 05917bf47aea42d397ccae57c3f29416 P 60ba3ffe3b7746469834cd91c84be14e: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:56:30.491932 16712 ts_tablet_manager.cc:1888] T 88137cb6ef4e4a6a96c5e8210fef0c96 P 76c386bef61a4bd5a907266a5118ffe2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:30.492008 16712 log.cc:1196] T 88137cb6ef4e4a6a96c5e8210fef0c96 P 76c386bef61a4bd5a907266a5118ffe2: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/wal/wals/88137cb6ef4e4a6a96c5e8210fef0c96
I20240503 13:56:30.492247 16712 ts_tablet_manager.cc:1909] T 88137cb6ef4e4a6a96c5e8210fef0c96 P 76c386bef61a4bd5a907266a5118ffe2: Deleting consensus metadata
I20240503 13:56:30.492365 16484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05917bf47aea42d397ccae57c3f29416. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:30.492434 16465 ts_tablet_manager.cc:902] T bb4f0bb47a29437090750c4486df2f02 P e47105a0f46e4511b7112e571688a4c7: Initiating tablet copy from peer ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:30.492897 16465 tablet_copy_client.cc:308] T bb4f0bb47a29437090750c4486df2f02 P e47105a0f46e4511b7112e571688a4c7: tablet copy: Beginning tablet copy session from remote peer at address 127.6.98.67:42279
I20240503 13:56:30.493098 12899 catalog_manager.cc:4830] TS 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885): tablet 88137cb6ef4e4a6a96c5e8210fef0c96 (table rebalancer_dynamic_table_new_name [id=0a1f605d9f474a4baf4693ac6e565608]) successfully deleted
I20240503 13:56:30.493590 13336 tablet_copy_service.cc:140] P ba0194098aa84449ae17e8a0d4c43703: Received BeginTabletCopySession request for tablet bb4f0bb47a29437090750c4486df2f02 from peer e47105a0f46e4511b7112e571688a4c7 ({username='slave'} at 127.6.98.72:58879)
I20240503 13:56:30.493686 13336 tablet_copy_service.cc:161] P ba0194098aa84449ae17e8a0d4c43703: Beginning new tablet copy session on tablet bb4f0bb47a29437090750c4486df2f02 from peer e47105a0f46e4511b7112e571688a4c7 at {username='slave'} at 127.6.98.72:58879: session id = e47105a0f46e4511b7112e571688a4c7-bb4f0bb47a29437090750c4486df2f02
I20240503 13:56:30.494465 16484 tablet_copy_client.cc:783] T 05917bf47aea42d397ccae57c3f29416 P 76c386bef61a4bd5a907266a5118ffe2: tablet copy: Starting download of 0 data blocks...
I20240503 13:56:30.494459 12911 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_extra_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:56:30.494572 13336 tablet_copy_source_session.cc:205] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:56:30.494603 16484 tablet_copy_client.cc:647] T 05917bf47aea42d397ccae57c3f29416 P 76c386bef61a4bd5a907266a5118ffe2: tablet copy: Starting download of 1 WAL segments...
I20240503 13:56:30.495059 15492 raft_consensus.cc:1203] T fd6a3d908840456f82c346c4bf1d8160 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:56:30.495424 16465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb4f0bb47a29437090750c4486df2f02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:30.497424 16465 tablet_copy_client.cc:783] T bb4f0bb47a29437090750c4486df2f02 P e47105a0f46e4511b7112e571688a4c7: tablet copy: Starting download of 0 data blocks...
I20240503 13:56:30.497566 16465 tablet_copy_client.cc:647] T bb4f0bb47a29437090750c4486df2f02 P e47105a0f46e4511b7112e571688a4c7: tablet copy: Starting download of 1 WAL segments...
I20240503 13:56:30.497741 16484 tablet_copy_client.cc:523] T 05917bf47aea42d397ccae57c3f29416 P 76c386bef61a4bd5a907266a5118ffe2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:56:30.499830 16484 tablet_bootstrap.cc:492] T 05917bf47aea42d397ccae57c3f29416 P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:30.502568 15738 tablet_service.cc:1467] Processing CreateTablet for tablet 6bca3e19362842fe928adb84e7c9297a (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=51ad326dfca444208bdd2a00d9582876]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:30.502807 15738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bca3e19362842fe928adb84e7c9297a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:30.504524 16734 tablet_bootstrap.cc:492] T 6bca3e19362842fe928adb84e7c9297a P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:30.504689 15737 tablet_service.cc:1467] Processing CreateTablet for tablet 3a140864f59b483bb5fea3de83d28ecf (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=51ad326dfca444208bdd2a00d9582876]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:30.504916 15737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a140864f59b483bb5fea3de83d28ecf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:30.505817 16734 tablet_bootstrap.cc:654] T 6bca3e19362842fe928adb84e7c9297a P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:30.506889 16734 tablet_bootstrap.cc:492] T 6bca3e19362842fe928adb84e7c9297a P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:30.506973 16734 ts_tablet_manager.cc:1367] T 6bca3e19362842fe928adb84e7c9297a P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:56:30.507412 16734 raft_consensus.cc:348] T 6bca3e19362842fe928adb84e7c9297a P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:30.507516 16734 raft_consensus.cc:374] T 6bca3e19362842fe928adb84e7c9297a P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:30.507550 16734 raft_consensus.cc:729] T 6bca3e19362842fe928adb84e7c9297a P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:30.507665 16734 consensus_queue.cc:260] T 6bca3e19362842fe928adb84e7c9297a P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:30.507922 16734 ts_tablet_manager.cc:1398] T 6bca3e19362842fe928adb84e7c9297a P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:30.507869 15604 tablet_service.cc:1467] Processing CreateTablet for tablet 6bca3e19362842fe928adb84e7c9297a (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=51ad326dfca444208bdd2a00d9582876]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:30.507963 15869 tablet_service.cc:1467] Processing CreateTablet for tablet 6bca3e19362842fe928adb84e7c9297a (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=51ad326dfca444208bdd2a00d9582876]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:30.508086 16734 tablet_bootstrap.cc:492] T 3a140864f59b483bb5fea3de83d28ecf P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:30.508128 15604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bca3e19362842fe928adb84e7c9297a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:30.508195 15869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bca3e19362842fe928adb84e7c9297a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:30.508322 15737 tablet_service.cc:1467] Processing CreateTablet for tablet 38596c626067458495846931e0ecb7eb (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=51ad326dfca444208bdd2a00d9582876]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:30.508550 15737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38596c626067458495846931e0ecb7eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:30.508921 16734 tablet_bootstrap.cc:654] T 3a140864f59b483bb5fea3de83d28ecf P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:30.509729 13034 tablet_service.cc:1467] Processing CreateTablet for tablet 38596c626067458495846931e0ecb7eb (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=51ad326dfca444208bdd2a00d9582876]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:30.509997 13034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38596c626067458495846931e0ecb7eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:30.510375 15868 tablet_service.cc:1467] Processing CreateTablet for tablet 3a140864f59b483bb5fea3de83d28ecf (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=51ad326dfca444208bdd2a00d9582876]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:30.510375 15605 tablet_service.cc:1467] Processing CreateTablet for tablet 3a140864f59b483bb5fea3de83d28ecf (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=51ad326dfca444208bdd2a00d9582876]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:30.510584 15868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a140864f59b483bb5fea3de83d28ecf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:30.510596 15605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a140864f59b483bb5fea3de83d28ecf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:30.510955 16465 tablet_copy_client.cc:523] T bb4f0bb47a29437090750c4486df2f02 P e47105a0f46e4511b7112e571688a4c7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:56:30.512508 15867 tablet_service.cc:1467] Processing CreateTablet for tablet 38596c626067458495846931e0ecb7eb (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=51ad326dfca444208bdd2a00d9582876]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:30.512622 16465 tablet_bootstrap.cc:492] T bb4f0bb47a29437090750c4486df2f02 P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:30.512760 15867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38596c626067458495846931e0ecb7eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:30.517741 16739 tablet_bootstrap.cc:492] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:30.518641 16739 tablet_bootstrap.cc:654] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:30.521095 16734 tablet_bootstrap.cc:492] T 3a140864f59b483bb5fea3de83d28ecf P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:30.521181 16734 ts_tablet_manager.cc:1367] T 3a140864f59b483bb5fea3de83d28ecf P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.002s
I20240503 13:56:30.521648 16734 raft_consensus.cc:348] T 3a140864f59b483bb5fea3de83d28ecf P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:30.521752 16734 raft_consensus.cc:374] T 3a140864f59b483bb5fea3de83d28ecf P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:30.521780 16734 raft_consensus.cc:729] T 3a140864f59b483bb5fea3de83d28ecf P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:30.521888 16734 consensus_queue.cc:260] T 3a140864f59b483bb5fea3de83d28ecf P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:30.522166 16734 ts_tablet_manager.cc:1398] T 3a140864f59b483bb5fea3de83d28ecf P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:30.522316 16734 tablet_bootstrap.cc:492] T 38596c626067458495846931e0ecb7eb P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:30.523314 16734 tablet_bootstrap.cc:654] T 38596c626067458495846931e0ecb7eb P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:30.525540 16734 tablet_bootstrap.cc:492] T 38596c626067458495846931e0ecb7eb P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:30.525684 16734 ts_tablet_manager.cc:1367] T 38596c626067458495846931e0ecb7eb P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 13:56:30.526139 16737 tablet_bootstrap.cc:492] T 6bca3e19362842fe928adb84e7c9297a P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:30.526155 16734 raft_consensus.cc:348] T 38596c626067458495846931e0ecb7eb P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:30.526278 16734 raft_consensus.cc:374] T 38596c626067458495846931e0ecb7eb P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:30.526314 16734 raft_consensus.cc:729] T 38596c626067458495846931e0ecb7eb P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:30.526441 16734 consensus_queue.cc:260] T 38596c626067458495846931e0ecb7eb P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:30.526729 16734 ts_tablet_manager.cc:1398] T 38596c626067458495846931e0ecb7eb P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:30.529757 16484 tablet_bootstrap.cc:492] T 05917bf47aea42d397ccae57c3f29416 P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:56:30.529997 16484 tablet_bootstrap.cc:492] T 05917bf47aea42d397ccae57c3f29416 P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap complete.
I20240503 13:56:30.530165 16484 ts_tablet_manager.cc:1367] T 05917bf47aea42d397ccae57c3f29416 P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.030s user 0.023s sys 0.000s
I20240503 13:56:30.530177 16740 ts_tablet_manager.cc:902] T 7a1c36d4fd0049139c98db7af595bc62 P e47105a0f46e4511b7112e571688a4c7: Initiating tablet copy from peer 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:30.530589 16484 raft_consensus.cc:348] T 05917bf47aea42d397ccae57c3f29416 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:30.530673 16739 tablet_bootstrap.cc:492] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:30.530712 16484 raft_consensus.cc:729] T 05917bf47aea42d397ccae57c3f29416 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:30.530764 16739 ts_tablet_manager.cc:1367] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 13:56:30.530839 16737 tablet_bootstrap.cc:654] T 6bca3e19362842fe928adb84e7c9297a P e47105a0f46e4511b7112e571688a4c7: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:30.530844 16484 consensus_queue.cc:260] T 05917bf47aea42d397ccae57c3f29416 P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:30.531121 16484 ts_tablet_manager.cc:1398] T 05917bf47aea42d397ccae57c3f29416 P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:30.531194 16739 raft_consensus.cc:348] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:30.531371 16739 raft_consensus.cc:374] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:30.531463 16739 raft_consensus.cc:729] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:30.531647 13074 tablet_copy_service.cc:342] P 60ba3ffe3b7746469834cd91c84be14e: Request end of tablet copy session 76c386bef61a4bd5a907266a5118ffe2-05917bf47aea42d397ccae57c3f29416 received from {username='slave'} at 127.6.98.71:42057
I20240503 13:56:30.531726 13074 tablet_copy_service.cc:433] P 60ba3ffe3b7746469834cd91c84be14e: ending tablet copy session 76c386bef61a4bd5a907266a5118ffe2-05917bf47aea42d397ccae57c3f29416 on tablet 05917bf47aea42d397ccae57c3f29416 with peer 76c386bef61a4bd5a907266a5118ffe2
I20240503 13:56:30.531715 16740 tablet_copy_client.cc:308] T 7a1c36d4fd0049139c98db7af595bc62 P e47105a0f46e4511b7112e571688a4c7: tablet copy: Beginning tablet copy session from remote peer at address 127.6.98.65:34695
I20240503 13:56:30.531679 16739 consensus_queue.cc:260] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:30.532269 16738 tablet_bootstrap.cc:492] T 6bca3e19362842fe928adb84e7c9297a P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:30.532977 13074 tablet_copy_service.cc:140] P 60ba3ffe3b7746469834cd91c84be14e: Received BeginTabletCopySession request for tablet 7a1c36d4fd0049139c98db7af595bc62 from peer e47105a0f46e4511b7112e571688a4c7 ({username='slave'} at 127.6.98.72:59759)
I20240503 13:56:30.533073 13074 tablet_copy_service.cc:161] P 60ba3ffe3b7746469834cd91c84be14e: Beginning new tablet copy session on tablet 7a1c36d4fd0049139c98db7af595bc62 from peer e47105a0f46e4511b7112e571688a4c7 at {username='slave'} at 127.6.98.72:59759: session id = e47105a0f46e4511b7112e571688a4c7-7a1c36d4fd0049139c98db7af595bc62
I20240503 13:56:30.533275 16737 tablet_bootstrap.cc:492] T 6bca3e19362842fe928adb84e7c9297a P e47105a0f46e4511b7112e571688a4c7: No bootstrap required, opened a new log
I20240503 13:56:30.532974 16739 ts_tablet_manager.cc:1398] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:30.533505 16737 ts_tablet_manager.cc:1367] T 6bca3e19362842fe928adb84e7c9297a P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 13:56:30.534021 13074 tablet_copy_source_session.cc:205] T 7a1c36d4fd0049139c98db7af595bc62 P 60ba3ffe3b7746469834cd91c84be14e: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:56:30.534411 15758 raft_consensus.cc:1203] T b47b28a4766a425e8b9bdfea882f979d P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:56:30.534730 16738 tablet_bootstrap.cc:654] T 6bca3e19362842fe928adb84e7c9297a P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:30.534766 16737 raft_consensus.cc:348] T 6bca3e19362842fe928adb84e7c9297a P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:30.534868 16737 raft_consensus.cc:374] T 6bca3e19362842fe928adb84e7c9297a P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:30.534899 16737 raft_consensus.cc:729] T 6bca3e19362842fe928adb84e7c9297a P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:30.535284 16737 consensus_queue.cc:260] T 6bca3e19362842fe928adb84e7c9297a P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:30.535584 16740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a1c36d4fd0049139c98db7af595bc62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:30.537257 16737 ts_tablet_manager.cc:1398] T 6bca3e19362842fe928adb84e7c9297a P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:56:30.537994 16737 tablet_bootstrap.cc:492] T 3a140864f59b483bb5fea3de83d28ecf P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:30.538676 16558 tablet_bootstrap.cc:492] T 866a258ecb2e4685908d5382cb594959 P 009f849ee7b54c80a3f5264695047e4d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:56:30.538926 16558 tablet_bootstrap.cc:492] T 866a258ecb2e4685908d5382cb594959 P 009f849ee7b54c80a3f5264695047e4d: Bootstrap complete.
I20240503 13:56:30.539122 16558 ts_tablet_manager.cc:1367] T 866a258ecb2e4685908d5382cb594959 P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.055s user 0.023s sys 0.000s
I20240503 13:56:30.539151 16737 tablet_bootstrap.cc:654] T 3a140864f59b483bb5fea3de83d28ecf P e47105a0f46e4511b7112e571688a4c7: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:30.539582 16558 raft_consensus.cc:348] T 866a258ecb2e4685908d5382cb594959 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:30.539708 16558 raft_consensus.cc:729] T 866a258ecb2e4685908d5382cb594959 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:30.540284 16738 tablet_bootstrap.cc:492] T 6bca3e19362842fe928adb84e7c9297a P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:30.540411 16738 ts_tablet_manager.cc:1367] T 6bca3e19362842fe928adb84e7c9297a P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 13:56:30.541319 16738 raft_consensus.cc:348] T 6bca3e19362842fe928adb84e7c9297a P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:30.541493 16738 raft_consensus.cc:374] T 6bca3e19362842fe928adb84e7c9297a P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:30.541574 16738 raft_consensus.cc:729] T 6bca3e19362842fe928adb84e7c9297a P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:30.541936 16740 tablet_copy_client.cc:783] T 7a1c36d4fd0049139c98db7af595bc62 P e47105a0f46e4511b7112e571688a4c7: tablet copy: Starting download of 0 data blocks...
I20240503 13:56:30.541901 16738 consensus_queue.cc:260] T 6bca3e19362842fe928adb84e7c9297a P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:30.542384 16738 ts_tablet_manager.cc:1398] T 6bca3e19362842fe928adb84e7c9297a P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:30.542582 16738 tablet_bootstrap.cc:492] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:30.543740 16740 tablet_copy_client.cc:647] T 7a1c36d4fd0049139c98db7af595bc62 P e47105a0f46e4511b7112e571688a4c7: tablet copy: Starting download of 1 WAL segments...
I20240503 13:56:30.543778 16738 tablet_bootstrap.cc:654] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:30.544420 16558 consensus_queue.cc:260] T 866a258ecb2e4685908d5382cb594959 P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:30.544879 16558 ts_tablet_manager.cc:1398] T 866a258ecb2e4685908d5382cb594959 P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240503 13:56:30.546005 16740 tablet_copy_client.cc:523] T 7a1c36d4fd0049139c98db7af595bc62 P e47105a0f46e4511b7112e571688a4c7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:56:30.548107 13467 tablet_copy_service.cc:342] P 7690ee79577c45e3a2c8f8b552c0bff8: Request end of tablet copy session 009f849ee7b54c80a3f5264695047e4d-866a258ecb2e4685908d5382cb594959 received from {username='slave'} at 127.6.98.70:40885
I20240503 13:56:30.548195 13467 tablet_copy_service.cc:433] P 7690ee79577c45e3a2c8f8b552c0bff8: ending tablet copy session 009f849ee7b54c80a3f5264695047e4d-866a258ecb2e4685908d5382cb594959 on tablet 866a258ecb2e4685908d5382cb594959 with peer 009f849ee7b54c80a3f5264695047e4d
I20240503 13:56:30.549695 16738 tablet_bootstrap.cc:492] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:30.549784 16738 ts_tablet_manager.cc:1367] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 13:56:30.550503 16248 raft_consensus.cc:482] T 6bca3e19362842fe928adb84e7c9297a P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:30.550582 16248 raft_consensus.cc:504] T 6bca3e19362842fe928adb84e7c9297a P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:30.551421 16738 raft_consensus.cc:348] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:30.553452 16738 raft_consensus.cc:374] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:30.553509 16465 tablet_bootstrap.cc:492] T bb4f0bb47a29437090750c4486df2f02 P e47105a0f46e4511b7112e571688a4c7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:56:30.552172 16248 leader_election.cc:288] T 6bca3e19362842fe928adb84e7c9297a P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:30.553550 16738 raft_consensus.cc:729] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:30.553777 16465 tablet_bootstrap.cc:492] T bb4f0bb47a29437090750c4486df2f02 P e47105a0f46e4511b7112e571688a4c7: Bootstrap complete.
I20240503 13:56:30.553947 16738 consensus_queue.cc:260] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:30.554289 16740 tablet_bootstrap.cc:492] T 7a1c36d4fd0049139c98db7af595bc62 P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:30.554293 16738 ts_tablet_manager.cc:1398] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 13:56:30.554914 16540 ts_tablet_manager.cc:902] T 3bb148fcf60f410089420a9499f08086 P 61b887f0d40d497badeca5604c1f0546: Initiating tablet copy from peer ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:30.555367 16540 tablet_copy_client.cc:308] T 3bb148fcf60f410089420a9499f08086 P 61b887f0d40d497badeca5604c1f0546: tablet copy: Beginning tablet copy session from remote peer at address 127.6.98.67:42279
I20240503 13:56:30.556279 13336 tablet_copy_service.cc:140] P ba0194098aa84449ae17e8a0d4c43703: Received BeginTabletCopySession request for tablet 3bb148fcf60f410089420a9499f08086 from peer 61b887f0d40d497badeca5604c1f0546 ({username='slave'} at 127.6.98.69:36581)
I20240503 13:56:30.556380 13336 tablet_copy_service.cc:161] P ba0194098aa84449ae17e8a0d4c43703: Beginning new tablet copy session on tablet 3bb148fcf60f410089420a9499f08086 from peer 61b887f0d40d497badeca5604c1f0546 at {username='slave'} at 127.6.98.69:36581: session id = 61b887f0d40d497badeca5604c1f0546-3bb148fcf60f410089420a9499f08086
I20240503 13:56:30.557336 13336 tablet_copy_source_session.cc:205] T 3bb148fcf60f410089420a9499f08086 P ba0194098aa84449ae17e8a0d4c43703: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:56:30.558036 16465 ts_tablet_manager.cc:1367] T bb4f0bb47a29437090750c4486df2f02 P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.045s user 0.023s sys 0.000s
I20240503 13:56:30.558202 16540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bb148fcf60f410089420a9499f08086. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:30.558462 16465 raft_consensus.cc:348] T bb4f0bb47a29437090750c4486df2f02 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:30.558622 16465 raft_consensus.cc:729] T bb4f0bb47a29437090750c4486df2f02 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:30.558990 16465 consensus_queue.cc:260] T bb4f0bb47a29437090750c4486df2f02 P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:30.560081 15624 raft_consensus.cc:1203] T 4647faf6a457415d8ae2b6691f452eab P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:56:30.560905 16465 ts_tablet_manager.cc:1398] T bb4f0bb47a29437090750c4486df2f02 P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:56:30.561311 13336 tablet_copy_service.cc:342] P ba0194098aa84449ae17e8a0d4c43703: Request end of tablet copy session e47105a0f46e4511b7112e571688a4c7-bb4f0bb47a29437090750c4486df2f02 received from {username='slave'} at 127.6.98.72:58879
I20240503 13:56:30.561383 13336 tablet_copy_service.cc:433] P ba0194098aa84449ae17e8a0d4c43703: ending tablet copy session e47105a0f46e4511b7112e571688a4c7-bb4f0bb47a29437090750c4486df2f02 on tablet bb4f0bb47a29437090750c4486df2f02 with peer e47105a0f46e4511b7112e571688a4c7
I20240503 13:56:30.561820 16540 tablet_copy_client.cc:783] T 3bb148fcf60f410089420a9499f08086 P 61b887f0d40d497badeca5604c1f0546: tablet copy: Starting download of 0 data blocks...
I20240503 13:56:30.561972 16540 tablet_copy_client.cc:647] T 3bb148fcf60f410089420a9499f08086 P 61b887f0d40d497badeca5604c1f0546: tablet copy: Starting download of 1 WAL segments...
I20240503 13:56:30.563910 15889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bca3e19362842fe928adb84e7c9297a" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7" is_pre_election: true
I20240503 13:56:30.564045 15889 raft_consensus.cc:2454] T 6bca3e19362842fe928adb84e7c9297a P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 0.
I20240503 13:56:30.564283 16737 tablet_bootstrap.cc:492] T 3a140864f59b483bb5fea3de83d28ecf P e47105a0f46e4511b7112e571688a4c7: No bootstrap required, opened a new log
I20240503 13:56:30.564414 16737 ts_tablet_manager.cc:1367] T 3a140864f59b483bb5fea3de83d28ecf P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.000s
I20240503 13:56:30.564441 15558 leader_election.cc:302] T 6bca3e19362842fe928adb84e7c9297a P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:30.564849 16248 raft_consensus.cc:2781] T 6bca3e19362842fe928adb84e7c9297a P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:30.564920 16248 raft_consensus.cc:482] T 6bca3e19362842fe928adb84e7c9297a P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:30.564953 16248 raft_consensus.cc:3037] T 6bca3e19362842fe928adb84e7c9297a P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:30.565950 16248 raft_consensus.cc:504] T 6bca3e19362842fe928adb84e7c9297a P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:30.566294 16540 tablet_copy_client.cc:523] T 3bb148fcf60f410089420a9499f08086 P 61b887f0d40d497badeca5604c1f0546: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:56:30.566313 16248 leader_election.cc:288] T 6bca3e19362842fe928adb84e7c9297a P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 election: Requested vote from peers 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:30.567726 16540 tablet_bootstrap.cc:492] T 3bb148fcf60f410089420a9499f08086 P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:30.569187 15889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bca3e19362842fe928adb84e7c9297a" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7"
I20240503 13:56:30.569305 15889 raft_consensus.cc:3037] T 6bca3e19362842fe928adb84e7c9297a P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:30.569317 16737 raft_consensus.cc:348] T 3a140864f59b483bb5fea3de83d28ecf P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:30.569430 16737 raft_consensus.cc:374] T 3a140864f59b483bb5fea3de83d28ecf P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:30.569461 16737 raft_consensus.cc:729] T 3a140864f59b483bb5fea3de83d28ecf P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:30.569609 16737 consensus_queue.cc:260] T 3a140864f59b483bb5fea3de83d28ecf P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:30.570190 15889 raft_consensus.cc:2454] T 6bca3e19362842fe928adb84e7c9297a P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 1.
I20240503 13:56:30.570758 15558 leader_election.cc:302] T 6bca3e19362842fe928adb84e7c9297a P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:30.570997 16248 raft_consensus.cc:2781] T 6bca3e19362842fe928adb84e7c9297a P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:30.571161 16248 raft_consensus.cc:686] T 6bca3e19362842fe928adb84e7c9297a P 009f849ee7b54c80a3f5264695047e4d [term 1 LEADER]: Becoming Leader. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Running, Role: LEADER
I20240503 13:56:30.571394 16248 consensus_queue.cc:237] T 6bca3e19362842fe928adb84e7c9297a P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:30.571672 16737 ts_tablet_manager.cc:1398] T 3a140864f59b483bb5fea3de83d28ecf P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240503 13:56:30.571806 15492 raft_consensus.cc:1203] T b291e3dd6a7a454ca4d736eff8dd3a41 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:56:30.573714 16737 tablet_bootstrap.cc:492] T 38596c626067458495846931e0ecb7eb P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:30.574668 16737 tablet_bootstrap.cc:654] T 38596c626067458495846931e0ecb7eb P e47105a0f46e4511b7112e571688a4c7: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:30.575001 12905 catalog_manager.cc:5484] T 6bca3e19362842fe928adb84e7c9297a P 009f849ee7b54c80a3f5264695047e4d reported cstate change: term changed from 0 to 1, leader changed from <none> to 009f849ee7b54c80a3f5264695047e4d (127.6.98.70). New cstate: current_term: 1 leader_uuid: "009f849ee7b54c80a3f5264695047e4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:30.576776 12904 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:56:30.583285 15758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bca3e19362842fe928adb84e7c9297a" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2" is_pre_election: true
I20240503 13:56:30.583461 15758 raft_consensus.cc:2454] T 6bca3e19362842fe928adb84e7c9297a P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 0.
I20240503 13:56:30.583640 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bca3e19362842fe928adb84e7c9297a" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2"
I20240503 13:56:30.583726 15757 raft_consensus.cc:3037] T 6bca3e19362842fe928adb84e7c9297a P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:30.584848 15757 raft_consensus.cc:2454] T 6bca3e19362842fe928adb84e7c9297a P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 1.
I20240503 13:56:30.587199 16737 tablet_bootstrap.cc:492] T 38596c626067458495846931e0ecb7eb P e47105a0f46e4511b7112e571688a4c7: No bootstrap required, opened a new log
I20240503 13:56:30.587388 16737 ts_tablet_manager.cc:1367] T 38596c626067458495846931e0ecb7eb P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240503 13:56:30.587873 15472 tablet_service.cc:1467] Processing CreateTablet for tablet 5e436c3a07b1401698c92cb0902c563a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c2d29cf742b7499da66c50002c53a288]), partition=
I20240503 13:56:30.588078 13427 tablet_service.cc:1467] Processing CreateTablet for tablet 5e436c3a07b1401698c92cb0902c563a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c2d29cf742b7499da66c50002c53a288]), partition=
I20240503 13:56:30.588179 15472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e436c3a07b1401698c92cb0902c563a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:30.588303 13427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e436c3a07b1401698c92cb0902c563a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:30.590926 15867 tablet_service.cc:1467] Processing CreateTablet for tablet 5e436c3a07b1401698c92cb0902c563a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c2d29cf742b7499da66c50002c53a288]), partition=
I20240503 13:56:30.591157 15867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e436c3a07b1401698c92cb0902c563a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:30.591985 16740 tablet_bootstrap.cc:492] T 7a1c36d4fd0049139c98db7af595bc62 P e47105a0f46e4511b7112e571688a4c7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:56:30.592222 16740 tablet_bootstrap.cc:492] T 7a1c36d4fd0049139c98db7af595bc62 P e47105a0f46e4511b7112e571688a4c7: Bootstrap complete.
I20240503 13:56:30.592366 16740 ts_tablet_manager.cc:1367] T 7a1c36d4fd0049139c98db7af595bc62 P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.038s user 0.024s sys 0.000s
I20240503 13:56:30.592813 16740 raft_consensus.cc:348] T 7a1c36d4fd0049139c98db7af595bc62 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:30.593039 16740 raft_consensus.cc:729] T 7a1c36d4fd0049139c98db7af595bc62 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:30.593235 16740 consensus_queue.cc:260] T 7a1c36d4fd0049139c98db7af595bc62 P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:30.593786 16737 raft_consensus.cc:348] T 38596c626067458495846931e0ecb7eb P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:30.594178 16737 raft_consensus.cc:374] T 38596c626067458495846931e0ecb7eb P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:30.594255 16764 tablet_bootstrap.cc:492] T 5e436c3a07b1401698c92cb0902c563a P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:30.594265 16737 raft_consensus.cc:729] T 38596c626067458495846931e0ecb7eb P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:30.594396 16737 consensus_queue.cc:260] T 38596c626067458495846931e0ecb7eb P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:30.594661 16737 ts_tablet_manager.cc:1398] T 38596c626067458495846931e0ecb7eb P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240503 13:56:30.594823 16737 tablet_bootstrap.cc:492] T 5e436c3a07b1401698c92cb0902c563a P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:30.595618 16737 tablet_bootstrap.cc:654] T 5e436c3a07b1401698c92cb0902c563a P e47105a0f46e4511b7112e571688a4c7: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:30.595861 16764 tablet_bootstrap.cc:654] T 5e436c3a07b1401698c92cb0902c563a P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:30.594041 16740 ts_tablet_manager.cc:1398] T 7a1c36d4fd0049139c98db7af595bc62 P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 13:56:30.596963 13074 tablet_copy_service.cc:342] P 60ba3ffe3b7746469834cd91c84be14e: Request end of tablet copy session e47105a0f46e4511b7112e571688a4c7-7a1c36d4fd0049139c98db7af595bc62 received from {username='slave'} at 127.6.98.72:59759
I20240503 13:56:30.597050 13074 tablet_copy_service.cc:433] P 60ba3ffe3b7746469834cd91c84be14e: ending tablet copy session e47105a0f46e4511b7112e571688a4c7-7a1c36d4fd0049139c98db7af595bc62 on tablet 7a1c36d4fd0049139c98db7af595bc62 with peer e47105a0f46e4511b7112e571688a4c7
I20240503 13:56:30.597224 16737 tablet_bootstrap.cc:492] T 5e436c3a07b1401698c92cb0902c563a P e47105a0f46e4511b7112e571688a4c7: No bootstrap required, opened a new log
I20240503 13:56:30.597527 16737 ts_tablet_manager.cc:1367] T 5e436c3a07b1401698c92cb0902c563a P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:56:30.598246 16763 tablet_bootstrap.cc:492] T 5e436c3a07b1401698c92cb0902c563a P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:30.598366 16737 raft_consensus.cc:348] T 5e436c3a07b1401698c92cb0902c563a P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:30.598480 16737 raft_consensus.cc:374] T 5e436c3a07b1401698c92cb0902c563a P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:30.598516 16737 raft_consensus.cc:729] T 5e436c3a07b1401698c92cb0902c563a P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:30.598626 16737 consensus_queue.cc:260] T 5e436c3a07b1401698c92cb0902c563a P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:30.598934 16737 ts_tablet_manager.cc:1398] T 5e436c3a07b1401698c92cb0902c563a P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:30.599942 16764 tablet_bootstrap.cc:492] T 5e436c3a07b1401698c92cb0902c563a P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:30.600035 16764 ts_tablet_manager.cc:1367] T 5e436c3a07b1401698c92cb0902c563a P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 13:56:30.600888 16764 raft_consensus.cc:348] T 5e436c3a07b1401698c92cb0902c563a P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:30.600997 16764 raft_consensus.cc:374] T 5e436c3a07b1401698c92cb0902c563a P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:30.601032 16764 raft_consensus.cc:729] T 5e436c3a07b1401698c92cb0902c563a P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:30.601143 16764 consensus_queue.cc:260] T 5e436c3a07b1401698c92cb0902c563a P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:30.601406 16764 ts_tablet_manager.cc:1398] T 5e436c3a07b1401698c92cb0902c563a P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:30.601920 16540 tablet_bootstrap.cc:492] T 3bb148fcf60f410089420a9499f08086 P 61b887f0d40d497badeca5604c1f0546: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:56:30.602177 16540 tablet_bootstrap.cc:492] T 3bb148fcf60f410089420a9499f08086 P 61b887f0d40d497badeca5604c1f0546: Bootstrap complete.
I20240503 13:56:30.602361 16540 ts_tablet_manager.cc:1367] T 3bb148fcf60f410089420a9499f08086 P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.035s user 0.025s sys 0.000s
I20240503 13:56:30.602769 16763 tablet_bootstrap.cc:654] T 5e436c3a07b1401698c92cb0902c563a P 61b887f0d40d497badeca5604c1f0546: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:30.602785 16540 raft_consensus.cc:348] T 3bb148fcf60f410089420a9499f08086 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:30.605176 16540 raft_consensus.cc:729] T 3bb148fcf60f410089420a9499f08086 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:30.605338 16540 consensus_queue.cc:260] T 3bb148fcf60f410089420a9499f08086 P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:30.605624 16540 ts_tablet_manager.cc:1398] T 3bb148fcf60f410089420a9499f08086 P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:56:30.606134 13336 tablet_copy_service.cc:342] P ba0194098aa84449ae17e8a0d4c43703: Request end of tablet copy session 61b887f0d40d497badeca5604c1f0546-3bb148fcf60f410089420a9499f08086 received from {username='slave'} at 127.6.98.69:36581
I20240503 13:56:30.606213 13336 tablet_copy_service.cc:433] P ba0194098aa84449ae17e8a0d4c43703: ending tablet copy session 61b887f0d40d497badeca5604c1f0546-3bb148fcf60f410089420a9499f08086 on tablet 3bb148fcf60f410089420a9499f08086 with peer 61b887f0d40d497badeca5604c1f0546
I20240503 13:56:30.612236 16763 tablet_bootstrap.cc:492] T 5e436c3a07b1401698c92cb0902c563a P 61b887f0d40d497badeca5604c1f0546: No bootstrap required, opened a new log
I20240503 13:56:30.612334 16763 ts_tablet_manager.cc:1367] T 5e436c3a07b1401698c92cb0902c563a P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.002s
I20240503 13:56:30.612807 16763 raft_consensus.cc:348] T 5e436c3a07b1401698c92cb0902c563a P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:30.613003 16763 raft_consensus.cc:374] T 5e436c3a07b1401698c92cb0902c563a P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:30.613054 16763 raft_consensus.cc:729] T 5e436c3a07b1401698c92cb0902c563a P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:30.613188 16763 consensus_queue.cc:260] T 5e436c3a07b1401698c92cb0902c563a P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:30.613528 16763 ts_tablet_manager.cc:1398] T 5e436c3a07b1401698c92cb0902c563a P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:30.637566 16484 ts_tablet_manager.cc:902] T e9b5dc0762494e6ebdebb7b2833a272e P 76c386bef61a4bd5a907266a5118ffe2: Initiating tablet copy from peer 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:30.638079 16484 tablet_copy_client.cc:308] T e9b5dc0762494e6ebdebb7b2833a272e P 76c386bef61a4bd5a907266a5118ffe2: tablet copy: Beginning tablet copy session from remote peer at address 127.6.98.65:34695
I20240503 13:56:30.638540 13074 tablet_copy_service.cc:140] P 60ba3ffe3b7746469834cd91c84be14e: Received BeginTabletCopySession request for tablet e9b5dc0762494e6ebdebb7b2833a272e from peer 76c386bef61a4bd5a907266a5118ffe2 ({username='slave'} at 127.6.98.71:42057)
I20240503 13:56:30.638648 13074 tablet_copy_service.cc:161] P 60ba3ffe3b7746469834cd91c84be14e: Beginning new tablet copy session on tablet e9b5dc0762494e6ebdebb7b2833a272e from peer 76c386bef61a4bd5a907266a5118ffe2 at {username='slave'} at 127.6.98.71:42057: session id = 76c386bef61a4bd5a907266a5118ffe2-e9b5dc0762494e6ebdebb7b2833a272e
I20240503 13:56:30.639402 13074 tablet_copy_source_session.cc:205] T e9b5dc0762494e6ebdebb7b2833a272e P 60ba3ffe3b7746469834cd91c84be14e: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:56:30.640475 16484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9b5dc0762494e6ebdebb7b2833a272e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:30.642582 16484 tablet_copy_client.cc:783] T e9b5dc0762494e6ebdebb7b2833a272e P 76c386bef61a4bd5a907266a5118ffe2: tablet copy: Starting download of 0 data blocks...
I20240503 13:56:30.642717 16484 tablet_copy_client.cc:647] T e9b5dc0762494e6ebdebb7b2833a272e P 76c386bef61a4bd5a907266a5118ffe2: tablet copy: Starting download of 1 WAL segments...
I20240503 13:56:30.653770 16484 tablet_copy_client.cc:523] T e9b5dc0762494e6ebdebb7b2833a272e P 76c386bef61a4bd5a907266a5118ffe2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:56:30.655226 16484 tablet_bootstrap.cc:492] T e9b5dc0762494e6ebdebb7b2833a272e P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:30.686395 15492 raft_consensus.cc:1203] T 9eb0d911701b4b2bb33a6b7ab59f08aa P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:56:30.697054 16484 tablet_bootstrap.cc:492] T e9b5dc0762494e6ebdebb7b2833a272e P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:56:30.697368 16484 tablet_bootstrap.cc:492] T e9b5dc0762494e6ebdebb7b2833a272e P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap complete.
I20240503 13:56:30.697417 16161 raft_consensus.cc:482] T 3a140864f59b483bb5fea3de83d28ecf P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:30.697566 16484 ts_tablet_manager.cc:1367] T e9b5dc0762494e6ebdebb7b2833a272e P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.042s user 0.018s sys 0.002s
I20240503 13:56:30.697537 16161 raft_consensus.cc:504] T 3a140864f59b483bb5fea3de83d28ecf P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:30.697904 16161 leader_election.cc:288] T 3a140864f59b483bb5fea3de83d28ecf P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951), 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885)
I20240503 13:56:30.698019 16484 raft_consensus.cc:348] T e9b5dc0762494e6ebdebb7b2833a272e P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:30.698243 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a140864f59b483bb5fea3de83d28ecf" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2" is_pre_election: true
I20240503 13:56:30.698292 16484 raft_consensus.cc:729] T e9b5dc0762494e6ebdebb7b2833a272e P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:30.698261 15624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a140864f59b483bb5fea3de83d28ecf" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d" is_pre_election: true
I20240503 13:56:30.698354 15757 raft_consensus.cc:2454] T 3a140864f59b483bb5fea3de83d28ecf P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 0.
I20240503 13:56:30.698392 15624 raft_consensus.cc:2454] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 0.
I20240503 13:56:30.698453 16484 consensus_queue.cc:260] T e9b5dc0762494e6ebdebb7b2833a272e P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:30.698621 15825 leader_election.cc:302] T 3a140864f59b483bb5fea3de83d28ecf P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76c386bef61a4bd5a907266a5118ffe2, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:30.698725 16484 ts_tablet_manager.cc:1398] T e9b5dc0762494e6ebdebb7b2833a272e P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 13:56:30.698999 16161 raft_consensus.cc:2781] T 3a140864f59b483bb5fea3de83d28ecf P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:30.699060 16161 raft_consensus.cc:482] T 3a140864f59b483bb5fea3de83d28ecf P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:30.699101 16161 raft_consensus.cc:3037] T 3a140864f59b483bb5fea3de83d28ecf P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:30.699190 13074 tablet_copy_service.cc:342] P 60ba3ffe3b7746469834cd91c84be14e: Request end of tablet copy session 76c386bef61a4bd5a907266a5118ffe2-e9b5dc0762494e6ebdebb7b2833a272e received from {username='slave'} at 127.6.98.71:42057
I20240503 13:56:30.699255 13074 tablet_copy_service.cc:433] P 60ba3ffe3b7746469834cd91c84be14e: ending tablet copy session 76c386bef61a4bd5a907266a5118ffe2-e9b5dc0762494e6ebdebb7b2833a272e on tablet e9b5dc0762494e6ebdebb7b2833a272e with peer 76c386bef61a4bd5a907266a5118ffe2
I20240503 13:56:30.699995 16161 raft_consensus.cc:504] T 3a140864f59b483bb5fea3de83d28ecf P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:30.700345 16161 leader_election.cc:288] T 3a140864f59b483bb5fea3de83d28ecf P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 election: Requested vote from peers 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951), 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885)
I20240503 13:56:30.700629 15624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a140864f59b483bb5fea3de83d28ecf" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d"
I20240503 13:56:30.700737 15624 raft_consensus.cc:3037] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:30.701057 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a140864f59b483bb5fea3de83d28ecf" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2"
I20240503 13:56:30.701153 15757 raft_consensus.cc:3037] T 3a140864f59b483bb5fea3de83d28ecf P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:30.701756 15624 raft_consensus.cc:2454] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 1.
I20240503 13:56:30.701921 15757 raft_consensus.cc:2454] T 3a140864f59b483bb5fea3de83d28ecf P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 1.
I20240503 13:56:30.702020 15822 leader_election.cc:302] T 3a140864f59b483bb5fea3de83d28ecf P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:30.702178 16161 raft_consensus.cc:2781] T 3a140864f59b483bb5fea3de83d28ecf P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:30.702256 16161 raft_consensus.cc:686] T 3a140864f59b483bb5fea3de83d28ecf P e47105a0f46e4511b7112e571688a4c7 [term 1 LEADER]: Becoming Leader. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Running, Role: LEADER
I20240503 13:56:30.702373 16161 consensus_queue.cc:237] T 3a140864f59b483bb5fea3de83d28ecf P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:30.703900 12906 catalog_manager.cc:5484] T 3a140864f59b483bb5fea3de83d28ecf P e47105a0f46e4511b7112e571688a4c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e47105a0f46e4511b7112e571688a4c7 (127.6.98.72). New cstate: current_term: 1 leader_uuid: "e47105a0f46e4511b7112e571688a4c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:30.755134 15332 raft_consensus.cc:482] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:30.755271 15332 raft_consensus.cc:504] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:30.755702 15332 leader_election.cc:288] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885)
I20240503 13:56:30.756093 15889 raft_consensus.cc:1203] T a746bcb2d718433f8c9acc8bd027a429 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:56:30.756175 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38596c626067458495846931e0ecb7eb" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2" is_pre_election: true
I20240503 13:56:30.756301 15757 raft_consensus.cc:2454] T 38596c626067458495846931e0ecb7eb P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 0.
I20240503 13:56:30.756902 12990 leader_election.cc:302] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 76c386bef61a4bd5a907266a5118ffe2; no voters:
I20240503 13:56:30.756042 15888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38596c626067458495846931e0ecb7eb" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7" is_pre_election: true
I20240503 13:56:30.757067 15332 raft_consensus.cc:2781] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:30.757133 15332 raft_consensus.cc:482] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:30.757174 15332 raft_consensus.cc:3037] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:30.757149 15888 raft_consensus.cc:2454] T 38596c626067458495846931e0ecb7eb P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 0.
I20240503 13:56:30.758039 15332 raft_consensus.cc:504] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:30.758421 15332 leader_election.cc:288] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 election: Requested vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885)
I20240503 13:56:30.758679 15888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38596c626067458495846931e0ecb7eb" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7"
I20240503 13:56:30.758790 15888 raft_consensus.cc:3037] T 38596c626067458495846931e0ecb7eb P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:30.758811 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38596c626067458495846931e0ecb7eb" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2"
I20240503 13:56:30.758987 15757 raft_consensus.cc:3037] T 38596c626067458495846931e0ecb7eb P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:30.759688 15888 raft_consensus.cc:2454] T 38596c626067458495846931e0ecb7eb P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 1.
I20240503 13:56:30.759845 15757 raft_consensus.cc:2454] T 38596c626067458495846931e0ecb7eb P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 1.
I20240503 13:56:30.760010 12987 leader_election.cc:302] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:30.760154 15332 raft_consensus.cc:2781] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:30.760241 15332 raft_consensus.cc:686] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e [term 1 LEADER]: Becoming Leader. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Running, Role: LEADER
I20240503 13:56:30.760392 15332 consensus_queue.cc:237] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:30.763578 12906 catalog_manager.cc:5484] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e reported cstate change: term changed from 0 to 1, leader changed from <none> to 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65). New cstate: current_term: 1 leader_uuid: "60ba3ffe3b7746469834cd91c84be14e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } }
I20240503 13:56:30.783262 16404 raft_consensus.cc:482] T 5e436c3a07b1401698c92cb0902c563a P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:30.783407 16404 raft_consensus.cc:504] T 5e436c3a07b1401698c92cb0902c563a P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:30.783947 16404 leader_election.cc:288] T 5e436c3a07b1401698c92cb0902c563a P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153)
I20240503 13:56:30.788609 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e436c3a07b1401698c92cb0902c563a" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
I20240503 13:56:30.788756 13446 raft_consensus.cc:2454] T 5e436c3a07b1401698c92cb0902c563a P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 0.
I20240503 13:56:30.789002 15822 leader_election.cc:302] T 5e436c3a07b1401698c92cb0902c563a P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:30.789044 15492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e436c3a07b1401698c92cb0902c563a" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546" is_pre_election: true
I20240503 13:56:30.789160 15492 raft_consensus.cc:2454] T 5e436c3a07b1401698c92cb0902c563a P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 0.
I20240503 13:56:30.789706 16404 raft_consensus.cc:2781] T 5e436c3a07b1401698c92cb0902c563a P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:30.789774 16404 raft_consensus.cc:482] T 5e436c3a07b1401698c92cb0902c563a P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:30.789815 16404 raft_consensus.cc:3037] T 5e436c3a07b1401698c92cb0902c563a P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:30.790041 15757 raft_consensus.cc:1203] T 1b714fbd920a447f866b8d4547275675 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:56:30.790849 16404 raft_consensus.cc:504] T 5e436c3a07b1401698c92cb0902c563a P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:30.791193 16404 leader_election.cc:288] T 5e436c3a07b1401698c92cb0902c563a P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 election: Requested vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153)
I20240503 13:56:30.791749 15492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e436c3a07b1401698c92cb0902c563a" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546"
I20240503 13:56:30.791874 15492 raft_consensus.cc:3037] T 5e436c3a07b1401698c92cb0902c563a P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:30.792678 15492 raft_consensus.cc:2454] T 5e436c3a07b1401698c92cb0902c563a P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 1.
I20240503 13:56:30.792961 15825 leader_election.cc:302] T 5e436c3a07b1401698c92cb0902c563a P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61b887f0d40d497badeca5604c1f0546, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:30.793128 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e436c3a07b1401698c92cb0902c563a" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
I20240503 13:56:30.793238 13446 raft_consensus.cc:3037] T 5e436c3a07b1401698c92cb0902c563a P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:30.793257 16404 raft_consensus.cc:2781] T 5e436c3a07b1401698c92cb0902c563a P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:30.793344 16404 raft_consensus.cc:686] T 5e436c3a07b1401698c92cb0902c563a P e47105a0f46e4511b7112e571688a4c7 [term 1 LEADER]: Becoming Leader. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Running, Role: LEADER
I20240503 13:56:30.793493 16404 consensus_queue.cc:237] T 5e436c3a07b1401698c92cb0902c563a P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:30.794127 13446 raft_consensus.cc:2454] T 5e436c3a07b1401698c92cb0902c563a P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 1.
I20240503 13:56:30.795540 12906 catalog_manager.cc:5484] T 5e436c3a07b1401698c92cb0902c563a P e47105a0f46e4511b7112e571688a4c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e47105a0f46e4511b7112e571688a4c7 (127.6.98.72). New cstate: current_term: 1 leader_uuid: "e47105a0f46e4511b7112e571688a4c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:30.817838 13034 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:30.820869 15472 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:56:30.822376 15737 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:56:30.823716 15605 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:56:30.827585 13427 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 13:56:30.836676 13296 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 13:56:30.840762 15867 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:56:30.848407 13164 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 13:56:30.926292 12909 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalance_test_table_8" } modify_external_catalogs: true
I20240503 13:56:30.926451 12909 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalance_test_table_8" } modify_external_catalogs: true
I20240503 13:56:30.929476 12909 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Sending DeleteTablet for 3 replicas of tablet 973a44e737844ffb99dd3e168fcdc1d5
I20240503 13:56:30.929812 12909 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Sending DeleteTablet for 4 replicas of tablet 08e3096cdac34940a3ab0529cb76c3c7
I20240503 13:56:30.930110 12909 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Sending DeleteTablet for 3 replicas of tablet e8e52f0e9d3340b2bd468e2c7d8bc921
I20240503 13:56:30.930083 13034 tablet_service.cc:1514] Processing DeleteTablet for tablet 973a44e737844ffb99dd3e168fcdc1d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:30 UTC) from {username='slave'} at 127.0.0.1:59458
I20240503 13:56:30.930284 16085 tablet_replica.cc:330] T 973a44e737844ffb99dd3e168fcdc1d5 P 60ba3ffe3b7746469834cd91c84be14e: stopping tablet replica
I20240503 13:56:30.930464 16085 raft_consensus.cc:2229] T 973a44e737844ffb99dd3e168fcdc1d5 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:30.930639 16085 raft_consensus.cc:2258] T 973a44e737844ffb99dd3e168fcdc1d5 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:30.930810 13423 tablet_service.cc:1514] Processing DeleteTablet for tablet e8e52f0e9d3340b2bd468e2c7d8bc921 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:30 UTC) from {username='slave'} at 127.0.0.1:33678
I20240503 13:56:30.930975 16087 tablet_replica.cc:330] T e8e52f0e9d3340b2bd468e2c7d8bc921 P 7690ee79577c45e3a2c8f8b552c0bff8: stopping tablet replica
I20240503 13:56:30.930974 12910 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:56:30.931084 16087 raft_consensus.cc:2229] T e8e52f0e9d3340b2bd468e2c7d8bc921 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:30.931195 16085 ts_tablet_manager.cc:1875] T 973a44e737844ffb99dd3e168fcdc1d5 P 60ba3ffe3b7746469834cd91c84be14e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:30.931232 16087 raft_consensus.cc:2258] T e8e52f0e9d3340b2bd468e2c7d8bc921 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:30.931720 16087 ts_tablet_manager.cc:1875] T e8e52f0e9d3340b2bd468e2c7d8bc921 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:30.931720 13165 tablet_service.cc:1514] Processing DeleteTablet for tablet 08e3096cdac34940a3ab0529cb76c3c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:30 UTC) from {username='slave'} at 127.0.0.1:41668
I20240503 13:56:30.933717 16087 ts_tablet_manager.cc:1888] T e8e52f0e9d3340b2bd468e2c7d8bc921 P 7690ee79577c45e3a2c8f8b552c0bff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:56:30.933776 16087 log.cc:1196] T e8e52f0e9d3340b2bd468e2c7d8bc921 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal/wals/e8e52f0e9d3340b2bd468e2c7d8bc921
I20240503 13:56:30.934087 13427 tablet_service.cc:1514] Processing DeleteTablet for tablet 08e3096cdac34940a3ab0529cb76c3c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:30 UTC) from {username='slave'} at 127.0.0.1:33678
I20240503 13:56:30.934206 13164 tablet_service.cc:1514] Processing DeleteTablet for tablet 973a44e737844ffb99dd3e168fcdc1d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:30 UTC) from {username='slave'} at 127.0.0.1:41668
I20240503 13:56:30.934260 15605 tablet_service.cc:1514] Processing DeleteTablet for tablet 08e3096cdac34940a3ab0529cb76c3c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:30 UTC) from {username='slave'} at 127.0.0.1:60718
I20240503 13:56:30.934708 13296 tablet_service.cc:1514] Processing DeleteTablet for tablet 973a44e737844ffb99dd3e168fcdc1d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:30 UTC) from {username='slave'} at 127.0.0.1:33250
I20240503 13:56:30.934976 16087 ts_tablet_manager.cc:1909] T e8e52f0e9d3340b2bd468e2c7d8bc921 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting consensus metadata
I20240503 13:56:30.935281 13295 tablet_service.cc:1514] Processing DeleteTablet for tablet e8e52f0e9d3340b2bd468e2c7d8bc921 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:30 UTC) from {username='slave'} at 127.0.0.1:33250
I20240503 13:56:30.936118 13033 tablet_service.cc:1514] Processing DeleteTablet for tablet e8e52f0e9d3340b2bd468e2c7d8bc921 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:30 UTC) from {username='slave'} at 127.0.0.1:59458
I20240503 13:56:30.936623 13034 tablet_service.cc:1514] Processing DeleteTablet for tablet 08e3096cdac34940a3ab0529cb76c3c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:30 UTC) from {username='slave'} at 127.0.0.1:59458
I20240503 13:56:30.937645 12896 catalog_manager.cc:4830] TS 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109): tablet e8e52f0e9d3340b2bd468e2c7d8bc921 (table rebalance_test_table_8 [id=575bc3a99f6c434980a3decff326d982]) successfully deleted
W20240503 13:56:30.937809 12896 catalog_manager.cc:4797] TS 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951): delete failed for tablet 08e3096cdac34940a3ab0529cb76c3c7 because the tablet was not found. No further retry: Not found: Tablet not found: 08e3096cdac34940a3ab0529cb76c3c7
I20240503 13:56:30.939332 16087 tablet_replica.cc:330] T 08e3096cdac34940a3ab0529cb76c3c7 P 7690ee79577c45e3a2c8f8b552c0bff8: stopping tablet replica
I20240503 13:56:30.939652 16087 raft_consensus.cc:2229] T 08e3096cdac34940a3ab0529cb76c3c7 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:30.940084 16087 raft_consensus.cc:2258] T 08e3096cdac34940a3ab0529cb76c3c7 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:30.940335 16085 ts_tablet_manager.cc:1888] T 973a44e737844ffb99dd3e168fcdc1d5 P 60ba3ffe3b7746469834cd91c84be14e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:56:30.940825 16085 log.cc:1196] T 973a44e737844ffb99dd3e168fcdc1d5 P 60ba3ffe3b7746469834cd91c84be14e: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal/wals/973a44e737844ffb99dd3e168fcdc1d5
I20240503 13:56:30.941195 16085 ts_tablet_manager.cc:1909] T 973a44e737844ffb99dd3e168fcdc1d5 P 60ba3ffe3b7746469834cd91c84be14e: Deleting consensus metadata
I20240503 13:56:30.942653 16085 tablet_replica.cc:330] T e8e52f0e9d3340b2bd468e2c7d8bc921 P 60ba3ffe3b7746469834cd91c84be14e: stopping tablet replica
I20240503 13:56:30.942848 16085 raft_consensus.cc:2229] T e8e52f0e9d3340b2bd468e2c7d8bc921 P 60ba3ffe3b7746469834cd91c84be14e [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:56:30.943150 16085 raft_consensus.cc:2258] T e8e52f0e9d3340b2bd468e2c7d8bc921 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:30.943192 15492 raft_consensus.cc:1261] T 5e436c3a07b1401698c92cb0902c563a P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Refusing update from remote peer e47105a0f46e4511b7112e571688a4c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:30.943792 16085 ts_tablet_manager.cc:1875] T e8e52f0e9d3340b2bd468e2c7d8bc921 P 60ba3ffe3b7746469834cd91c84be14e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:30.944555 16404 consensus_queue.cc:1035] T 5e436c3a07b1401698c92cb0902c563a P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:30.945264 12897 catalog_manager.cc:4830] TS 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695): tablet 973a44e737844ffb99dd3e168fcdc1d5 (table rebalance_test_table_8 [id=575bc3a99f6c434980a3decff326d982]) successfully deleted
I20240503 13:56:30.946458 16864 tablet_replica.cc:330] T 973a44e737844ffb99dd3e168fcdc1d5 P ba0194098aa84449ae17e8a0d4c43703: stopping tablet replica
I20240503 13:56:30.946595 16864 raft_consensus.cc:2229] T 973a44e737844ffb99dd3e168fcdc1d5 P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:56:30.946682 16085 ts_tablet_manager.cc:1888] T e8e52f0e9d3340b2bd468e2c7d8bc921 P 60ba3ffe3b7746469834cd91c84be14e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:56:30.946745 16085 log.cc:1196] T e8e52f0e9d3340b2bd468e2c7d8bc921 P 60ba3ffe3b7746469834cd91c84be14e: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal/wals/e8e52f0e9d3340b2bd468e2c7d8bc921
I20240503 13:56:30.946878 16864 raft_consensus.cc:2258] T 973a44e737844ffb99dd3e168fcdc1d5 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:30.946990 16085 ts_tablet_manager.cc:1909] T e8e52f0e9d3340b2bd468e2c7d8bc921 P 60ba3ffe3b7746469834cd91c84be14e: Deleting consensus metadata
I20240503 13:56:30.947414 16864 ts_tablet_manager.cc:1875] T 973a44e737844ffb99dd3e168fcdc1d5 P ba0194098aa84449ae17e8a0d4c43703: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:30.947727 12897 catalog_manager.cc:4830] TS 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695): tablet e8e52f0e9d3340b2bd468e2c7d8bc921 (table rebalance_test_table_8 [id=575bc3a99f6c434980a3decff326d982]) successfully deleted
I20240503 13:56:30.947765 16085 tablet_replica.cc:330] T 08e3096cdac34940a3ab0529cb76c3c7 P 60ba3ffe3b7746469834cd91c84be14e: stopping tablet replica
I20240503 13:56:30.947880 16085 raft_consensus.cc:2229] T 08e3096cdac34940a3ab0529cb76c3c7 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:30.948020 16085 raft_consensus.cc:2258] T 08e3096cdac34940a3ab0529cb76c3c7 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:30.945425 13446 raft_consensus.cc:1261] T 5e436c3a07b1401698c92cb0902c563a P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer e47105a0f46e4511b7112e571688a4c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:30.948803 16161 consensus_queue.cc:1035] T 5e436c3a07b1401698c92cb0902c563a P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:30.948900 16085 ts_tablet_manager.cc:1875] T 08e3096cdac34940a3ab0529cb76c3c7 P 60ba3ffe3b7746469834cd91c84be14e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:30.949457 16087 ts_tablet_manager.cc:1875] T 08e3096cdac34940a3ab0529cb76c3c7 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:30.949887 16864 ts_tablet_manager.cc:1888] T 973a44e737844ffb99dd3e168fcdc1d5 P ba0194098aa84449ae17e8a0d4c43703: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:56:30.949960 16864 log.cc:1196] T 973a44e737844ffb99dd3e168fcdc1d5 P ba0194098aa84449ae17e8a0d4c43703: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/wal/wals/973a44e737844ffb99dd3e168fcdc1d5
I20240503 13:56:30.950201 16864 ts_tablet_manager.cc:1909] T 973a44e737844ffb99dd3e168fcdc1d5 P ba0194098aa84449ae17e8a0d4c43703: Deleting consensus metadata
I20240503 13:56:30.951026 16864 tablet_replica.cc:330] T e8e52f0e9d3340b2bd468e2c7d8bc921 P ba0194098aa84449ae17e8a0d4c43703: stopping tablet replica
I20240503 13:56:30.951018 12899 catalog_manager.cc:4830] TS ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279): tablet 973a44e737844ffb99dd3e168fcdc1d5 (table rebalance_test_table_8 [id=575bc3a99f6c434980a3decff326d982]) successfully deleted
I20240503 13:56:30.951172 16864 raft_consensus.cc:2229] T e8e52f0e9d3340b2bd468e2c7d8bc921 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:30.951310 16864 raft_consensus.cc:2258] T e8e52f0e9d3340b2bd468e2c7d8bc921 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:30.948450 12906 catalog_manager.cc:5576] TS 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153) does not have the latest schema for tablet 5e436c3a07b1401698c92cb0902c563a (table rebalancer_dynamic_table [id=c2d29cf742b7499da66c50002c53a288]). Expected version 1 got 0
I20240503 13:56:30.951822 16864 ts_tablet_manager.cc:1875] T e8e52f0e9d3340b2bd468e2c7d8bc921 P ba0194098aa84449ae17e8a0d4c43703: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:30.952029 16085 ts_tablet_manager.cc:1888] T 08e3096cdac34940a3ab0529cb76c3c7 P 60ba3ffe3b7746469834cd91c84be14e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:30.952090 16085 log.cc:1196] T 08e3096cdac34940a3ab0529cb76c3c7 P 60ba3ffe3b7746469834cd91c84be14e: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal/wals/08e3096cdac34940a3ab0529cb76c3c7
I20240503 13:56:30.952327 16085 ts_tablet_manager.cc:1909] T 08e3096cdac34940a3ab0529cb76c3c7 P 60ba3ffe3b7746469834cd91c84be14e: Deleting consensus metadata
I20240503 13:56:30.952348 12909 catalog_manager.cc:5576] TS 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109) does not have the latest schema for tablet 5e436c3a07b1401698c92cb0902c563a (table rebalancer_dynamic_table [id=c2d29cf742b7499da66c50002c53a288]). Expected version 1 got 0
I20240503 13:56:30.953099 16087 ts_tablet_manager.cc:1888] T 08e3096cdac34940a3ab0529cb76c3c7 P 7690ee79577c45e3a2c8f8b552c0bff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:30.953202 16087 log.cc:1196] T 08e3096cdac34940a3ab0529cb76c3c7 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal/wals/08e3096cdac34940a3ab0529cb76c3c7
I20240503 13:56:30.953235 12897 catalog_manager.cc:4830] TS 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695): tablet 08e3096cdac34940a3ab0529cb76c3c7 (table rebalance_test_table_8 [id=575bc3a99f6c434980a3decff326d982]) successfully deleted
I20240503 13:56:30.953511 16087 ts_tablet_manager.cc:1909] T 08e3096cdac34940a3ab0529cb76c3c7 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting consensus metadata
I20240503 13:56:30.953780 16864 ts_tablet_manager.cc:1888] T e8e52f0e9d3340b2bd468e2c7d8bc921 P ba0194098aa84449ae17e8a0d4c43703: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:56:30.953899 16864 log.cc:1196] T e8e52f0e9d3340b2bd468e2c7d8bc921 P ba0194098aa84449ae17e8a0d4c43703: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/wal/wals/e8e52f0e9d3340b2bd468e2c7d8bc921
I20240503 13:56:30.954206 16864 ts_tablet_manager.cc:1909] T e8e52f0e9d3340b2bd468e2c7d8bc921 P ba0194098aa84449ae17e8a0d4c43703: Deleting consensus metadata
I20240503 13:56:30.955456 12899 catalog_manager.cc:4830] TS ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279): tablet e8e52f0e9d3340b2bd468e2c7d8bc921 (table rebalance_test_table_8 [id=575bc3a99f6c434980a3decff326d982]) successfully deleted
I20240503 13:56:30.955578 16869 tablet.cc:1722] T 5e436c3a07b1401698c92cb0902c563a P e47105a0f46e4511b7112e571688a4c7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:30.955917 12896 catalog_manager.cc:4830] TS 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109): tablet 08e3096cdac34940a3ab0529cb76c3c7 (table rebalance_test_table_8 [id=575bc3a99f6c434980a3decff326d982]) successfully deleted
I20240503 13:56:30.956050 16869 tablet.cc:1620] T 5e436c3a07b1401698c92cb0902c563a P e47105a0f46e4511b7112e571688a4c7: MemRowSet was empty: no flush needed.
I20240503 13:56:30.957309 12905 catalog_manager.cc:5576] TS 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109) does not have the latest schema for tablet 5e436c3a07b1401698c92cb0902c563a (table rebalancer_dynamic_table [id=c2d29cf742b7499da66c50002c53a288]). Expected version 1 got 0
I20240503 13:56:30.962366 12909 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=c2d29cf742b7499da66c50002c53a288] alter complete (version 1)
I20240503 13:56:30.966640 16863 tablet_replica.cc:330] T 08e3096cdac34940a3ab0529cb76c3c7 P 8ccf5a4ac450414c9d3579bd5800b770: stopping tablet replica
I20240503 13:56:30.966815 16863 raft_consensus.cc:2229] T 08e3096cdac34940a3ab0529cb76c3c7 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:56:30.967098 16863 raft_consensus.cc:2258] T 08e3096cdac34940a3ab0529cb76c3c7 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:30.967968 16863 ts_tablet_manager.cc:1875] T 08e3096cdac34940a3ab0529cb76c3c7 P 8ccf5a4ac450414c9d3579bd5800b770: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:30.968911 16865 mvcc.cc:205] Tried to move back new op lower bound from 7023593844497866752 to 7023593843891421184. Current Snapshot: MvccSnapshot[applied={T|T < 7023593844497866752}]
I20240503 13:56:30.969286 16871 tablet.cc:1722] T 5e436c3a07b1401698c92cb0902c563a P 61b887f0d40d497badeca5604c1f0546: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:30.969990 12906 catalog_manager.cc:5576] TS 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153) does not have the latest schema for tablet 5e436c3a07b1401698c92cb0902c563a (table rebalancer_dynamic_table [id=c2d29cf742b7499da66c50002c53a288]). Expected version 1 got 0
I20240503 13:56:30.970468 16871 tablet.cc:1620] T 5e436c3a07b1401698c92cb0902c563a P 61b887f0d40d497badeca5604c1f0546: MemRowSet was empty: no flush needed.
I20240503 13:56:30.970712 16863 ts_tablet_manager.cc:1888] T 08e3096cdac34940a3ab0529cb76c3c7 P 8ccf5a4ac450414c9d3579bd5800b770: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:30.970782 16863 log.cc:1196] T 08e3096cdac34940a3ab0529cb76c3c7 P 8ccf5a4ac450414c9d3579bd5800b770: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/wal/wals/08e3096cdac34940a3ab0529cb76c3c7
I20240503 13:56:30.971036 16863 ts_tablet_manager.cc:1909] T 08e3096cdac34940a3ab0529cb76c3c7 P 8ccf5a4ac450414c9d3579bd5800b770: Deleting consensus metadata
I20240503 13:56:30.971987 16863 tablet_replica.cc:330] T 973a44e737844ffb99dd3e168fcdc1d5 P 8ccf5a4ac450414c9d3579bd5800b770: stopping tablet replica
I20240503 13:56:30.972033 12899 catalog_manager.cc:4830] TS 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387): tablet 08e3096cdac34940a3ab0529cb76c3c7 (table rebalance_test_table_8 [id=575bc3a99f6c434980a3decff326d982]) successfully deleted
I20240503 13:56:30.972128 16863 raft_consensus.cc:2229] T 973a44e737844ffb99dd3e168fcdc1d5 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:30.972251 16863 raft_consensus.cc:2258] T 973a44e737844ffb99dd3e168fcdc1d5 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:30.972847 16863 ts_tablet_manager.cc:1875] T 973a44e737844ffb99dd3e168fcdc1d5 P 8ccf5a4ac450414c9d3579bd5800b770: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:30.967386 16870 tablet.cc:1722] T 5e436c3a07b1401698c92cb0902c563a P 7690ee79577c45e3a2c8f8b552c0bff8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:30.973788 16870 tablet.cc:1620] T 5e436c3a07b1401698c92cb0902c563a P 7690ee79577c45e3a2c8f8b552c0bff8: MemRowSet was empty: no flush needed.
I20240503 13:56:30.976228 16863 ts_tablet_manager.cc:1888] T 973a44e737844ffb99dd3e168fcdc1d5 P 8ccf5a4ac450414c9d3579bd5800b770: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:56:30.976315 16863 log.cc:1196] T 973a44e737844ffb99dd3e168fcdc1d5 P 8ccf5a4ac450414c9d3579bd5800b770: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/wal/wals/973a44e737844ffb99dd3e168fcdc1d5
I20240503 13:56:30.976579 16863 ts_tablet_manager.cc:1909] T 973a44e737844ffb99dd3e168fcdc1d5 P 8ccf5a4ac450414c9d3579bd5800b770: Deleting consensus metadata
I20240503 13:56:30.977252 12899 catalog_manager.cc:4830] TS 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387): tablet 973a44e737844ffb99dd3e168fcdc1d5 (table rebalance_test_table_8 [id=575bc3a99f6c434980a3decff326d982]) successfully deleted
I20240503 13:56:30.978401 16759 consensus_queue.cc:1035] T 6bca3e19362842fe928adb84e7c9297a P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:30.983867 16248 consensus_queue.cc:1035] T 6bca3e19362842fe928adb84e7c9297a P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:30.995833 15472 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:56:30.996455 15605 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:56:30.998041 13034 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:31.004704 13427 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 13:56:31.014299 15867 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:56:31.015733 15492 raft_consensus.cc:1203] T 3bb148fcf60f410089420a9499f08086 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:56:31.022812 15737 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:56:31.030953 13164 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 13:56:31.033761 13295 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 13:56:31.049598 15888 raft_consensus.cc:1203] T 7a1c36d4fd0049139c98db7af595bc62 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:56:31.059533 15757 raft_consensus.cc:1203] T 05917bf47aea42d397ccae57c3f29416 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:56:31.073346 12905 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:56:31.085584 16869 tablet.cc:1722] T 5e436c3a07b1401698c92cb0902c563a P e47105a0f46e4511b7112e571688a4c7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:31.085942 16869 tablet.cc:1620] T 5e436c3a07b1401698c92cb0902c563a P e47105a0f46e4511b7112e571688a4c7: MemRowSet was empty: no flush needed.
I20240503 13:56:31.085958 16870 tablet.cc:1722] T 5e436c3a07b1401698c92cb0902c563a P 7690ee79577c45e3a2c8f8b552c0bff8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:31.086136 16871 tablet.cc:1722] T 5e436c3a07b1401698c92cb0902c563a P 61b887f0d40d497badeca5604c1f0546: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:31.086262 16870 tablet.cc:1620] T 5e436c3a07b1401698c92cb0902c563a P 7690ee79577c45e3a2c8f8b552c0bff8: MemRowSet was empty: no flush needed.
I20240503 13:56:31.086442 16871 tablet.cc:1620] T 5e436c3a07b1401698c92cb0902c563a P 61b887f0d40d497badeca5604c1f0546: MemRowSet was empty: no flush needed.
I20240503 13:56:31.091727 15624 raft_consensus.cc:1203] T 866a258ecb2e4685908d5382cb594959 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:56:31.092752 12909 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=c2d29cf742b7499da66c50002c53a288] alter complete (version 2)
I20240503 13:56:31.100917 15888 raft_consensus.cc:1203] T bb4f0bb47a29437090750c4486df2f02 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:56:31.136227 15757 raft_consensus.cc:1203] T e9b5dc0762494e6ebdebb7b2833a272e P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:56:31.144878 15958 consensus_queue.cc:1035] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:31.151319 15958 consensus_queue.cc:1035] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:31.155614 13034 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:31.162598 15472 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:56:31.163604 15737 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:56:31.164796 15605 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:56:31.165071 13295 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 13:56:31.166886 13427 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 13:56:31.184937 13164 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 13:56:31.194584 12904 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:56:31.202607 16869 tablet.cc:1722] T 5e436c3a07b1401698c92cb0902c563a P e47105a0f46e4511b7112e571688a4c7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:31.203084 16161 consensus_queue.cc:1035] T 3a140864f59b483bb5fea3de83d28ecf P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:31.203109 16870 tablet.cc:1722] T 5e436c3a07b1401698c92cb0902c563a P 7690ee79577c45e3a2c8f8b552c0bff8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:31.203302 16871 tablet.cc:1722] T 5e436c3a07b1401698c92cb0902c563a P 61b887f0d40d497badeca5604c1f0546: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:31.205941 15867 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:56:31.210319 12906 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table_new_name [id=c2d29cf742b7499da66c50002c53a288] alter complete (version 3)
I20240503 13:56:31.212006 16404 consensus_queue.cc:1035] T 3a140864f59b483bb5fea3de83d28ecf P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:31.269905 13052 consensus_queue.cc:237] T 05917bf47aea42d397ccae57c3f29416 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:31.271134 13447 raft_consensus.cc:1261] T 05917bf47aea42d397ccae57c3f29416 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer 60ba3ffe3b7746469834cd91c84be14e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:56:31.271281 15757 raft_consensus.cc:1261] T 05917bf47aea42d397ccae57c3f29416 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Refusing update from remote peer 60ba3ffe3b7746469834cd91c84be14e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:56:31.271624 15958 consensus_queue.cc:1035] T 05917bf47aea42d397ccae57c3f29416 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:31.271771 15332 consensus_queue.cc:1035] T 05917bf47aea42d397ccae57c3f29416 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:31.273438 15958 raft_consensus.cc:2932] T 05917bf47aea42d397ccae57c3f29416 P 60ba3ffe3b7746469834cd91c84be14e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:31.275281 13447 raft_consensus.cc:2932] T 05917bf47aea42d397ccae57c3f29416 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:31.275688 15757 raft_consensus.cc:2932] T 05917bf47aea42d397ccae57c3f29416 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:31.277345 12906 catalog_manager.cc:5484] T 05917bf47aea42d397ccae57c3f29416 P 60ba3ffe3b7746469834cd91c84be14e reported cstate change: config changed from index 4 to 5, VOTER ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) evicted. New cstate: current_term: 1 leader_uuid: "60ba3ffe3b7746469834cd91c84be14e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:31.281991 13295 tablet_service.cc:1514] Processing DeleteTablet for tablet 05917bf47aea42d397ccae57c3f29416 with delete_type TABLET_DATA_TOMBSTONED (TS ba0194098aa84449ae17e8a0d4c43703 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:33250
I20240503 13:56:31.282356 16864 tablet_replica.cc:330] T 05917bf47aea42d397ccae57c3f29416 P ba0194098aa84449ae17e8a0d4c43703: stopping tablet replica
I20240503 13:56:31.282509 16864 raft_consensus.cc:2229] T 05917bf47aea42d397ccae57c3f29416 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:31.282655 16864 raft_consensus.cc:2258] T 05917bf47aea42d397ccae57c3f29416 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:31.283094 16864 ts_tablet_manager.cc:1875] T 05917bf47aea42d397ccae57c3f29416 P ba0194098aa84449ae17e8a0d4c43703: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:56:31.285065 16864 ts_tablet_manager.cc:1888] T 05917bf47aea42d397ccae57c3f29416 P ba0194098aa84449ae17e8a0d4c43703: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 13:56:31.285142 16864 log.cc:1196] T 05917bf47aea42d397ccae57c3f29416 P ba0194098aa84449ae17e8a0d4c43703: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/wal/wals/05917bf47aea42d397ccae57c3f29416
I20240503 13:56:31.285751 12899 catalog_manager.cc:4830] TS ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279): tablet 05917bf47aea42d397ccae57c3f29416 (table rebalance_test_table_11 [id=ba5c23cc430441c0a236a4359195fcb0]) successfully deleted
I20240503 13:56:31.318145 12909 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:31.318290 12909 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:31.320470 12909 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Sending DeleteTablet for 3 replicas of tablet 5e436c3a07b1401698c92cb0902c563a
I20240503 13:56:31.321161 13427 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e436c3a07b1401698c92cb0902c563a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:31 UTC) from {username='slave'} at 127.0.0.1:33678
I20240503 13:56:31.321782 16087 tablet_replica.cc:330] T 5e436c3a07b1401698c92cb0902c563a P 7690ee79577c45e3a2c8f8b552c0bff8: stopping tablet replica
I20240503 13:56:31.322003 16087 raft_consensus.cc:2229] T 5e436c3a07b1401698c92cb0902c563a P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:31.322149 16087 raft_consensus.cc:2258] T 5e436c3a07b1401698c92cb0902c563a P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:31.322408 15472 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e436c3a07b1401698c92cb0902c563a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:31 UTC) from {username='slave'} at 127.0.0.1:58758
I20240503 13:56:31.322810 15867 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e436c3a07b1401698c92cb0902c563a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:31 UTC) from {username='slave'} at 127.0.0.1:50498
I20240503 13:56:31.322897 16992 tablet_replica.cc:330] T 5e436c3a07b1401698c92cb0902c563a P 61b887f0d40d497badeca5604c1f0546: stopping tablet replica
I20240503 13:56:31.323381 16992 raft_consensus.cc:2229] T 5e436c3a07b1401698c92cb0902c563a P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:31.323602 16992 raft_consensus.cc:2258] T 5e436c3a07b1401698c92cb0902c563a P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:31.324213 12909 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_extra_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:56:31.324896 16992 ts_tablet_manager.cc:1875] T 5e436c3a07b1401698c92cb0902c563a P 61b887f0d40d497badeca5604c1f0546: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:31.327258 16087 ts_tablet_manager.cc:1875] T 5e436c3a07b1401698c92cb0902c563a P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:31.327812 16992 ts_tablet_manager.cc:1888] T 5e436c3a07b1401698c92cb0902c563a P 61b887f0d40d497badeca5604c1f0546: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:31.327904 16992 log.cc:1196] T 5e436c3a07b1401698c92cb0902c563a P 61b887f0d40d497badeca5604c1f0546: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/wal/wals/5e436c3a07b1401698c92cb0902c563a
I20240503 13:56:31.328207 16992 ts_tablet_manager.cc:1909] T 5e436c3a07b1401698c92cb0902c563a P 61b887f0d40d497badeca5604c1f0546: Deleting consensus metadata
I20240503 13:56:31.329479 16087 ts_tablet_manager.cc:1888] T 5e436c3a07b1401698c92cb0902c563a P 7690ee79577c45e3a2c8f8b552c0bff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:31.329470 12899 catalog_manager.cc:4830] TS 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153): tablet 5e436c3a07b1401698c92cb0902c563a (table rebalancer_dynamic_table_new_name [id=c2d29cf742b7499da66c50002c53a288]) successfully deleted
I20240503 13:56:31.329556 16087 log.cc:1196] T 5e436c3a07b1401698c92cb0902c563a P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal/wals/5e436c3a07b1401698c92cb0902c563a
I20240503 13:56:31.329792 16087 ts_tablet_manager.cc:1909] T 5e436c3a07b1401698c92cb0902c563a P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting consensus metadata
I20240503 13:56:31.331180 12896 catalog_manager.cc:4830] TS 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109): tablet 5e436c3a07b1401698c92cb0902c563a (table rebalancer_dynamic_table_new_name [id=c2d29cf742b7499da66c50002c53a288]) successfully deleted
I20240503 13:56:31.331528 16993 tablet_replica.cc:330] T 5e436c3a07b1401698c92cb0902c563a P e47105a0f46e4511b7112e571688a4c7: stopping tablet replica
I20240503 13:56:31.331672 16993 raft_consensus.cc:2229] T 5e436c3a07b1401698c92cb0902c563a P e47105a0f46e4511b7112e571688a4c7 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:56:31.332013 16993 raft_consensus.cc:2258] T 5e436c3a07b1401698c92cb0902c563a P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:31.332928 13034 tablet_service.cc:1467] Processing CreateTablet for tablet fdc5f76f63ec4528ad57202555caf243 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=1161fd279f6d435887536e9c124d251a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:31.333128 13033 tablet_service.cc:1467] Processing CreateTablet for tablet fcb0aa9fce544ae2a6f1ca96c5315c36 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=1161fd279f6d435887536e9c124d251a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:31.333225 13034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdc5f76f63ec4528ad57202555caf243. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:31.333415 13033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcb0aa9fce544ae2a6f1ca96c5315c36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:31.333956 15737 tablet_service.cc:1467] Processing CreateTablet for tablet fdc5f76f63ec4528ad57202555caf243 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=1161fd279f6d435887536e9c124d251a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:31.334100 15736 tablet_service.cc:1467] Processing CreateTablet for tablet fcb0aa9fce544ae2a6f1ca96c5315c36 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=1161fd279f6d435887536e9c124d251a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:31.334266 15737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdc5f76f63ec4528ad57202555caf243. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:31.334437 15736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcb0aa9fce544ae2a6f1ca96c5315c36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:31.334658 15738 tablet_service.cc:1467] Processing CreateTablet for tablet 604875eff3594876af54e9126057bd7f (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=1161fd279f6d435887536e9c124d251a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:31.334862 15738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 604875eff3594876af54e9126057bd7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:31.336241 15472 tablet_service.cc:1467] Processing CreateTablet for tablet fdc5f76f63ec4528ad57202555caf243 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=1161fd279f6d435887536e9c124d251a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:31.336532 15472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdc5f76f63ec4528ad57202555caf243. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:31.336737 17012 tablet_bootstrap.cc:492] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:31.338004 17012 tablet_bootstrap.cc:654] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:31.338912 15473 tablet_service.cc:1467] Processing CreateTablet for tablet 604875eff3594876af54e9126057bd7f (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=1161fd279f6d435887536e9c124d251a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:31.339172 15473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 604875eff3594876af54e9126057bd7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:31.339274 17012 tablet_bootstrap.cc:492] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:31.339367 17012 ts_tablet_manager.cc:1367] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:56:31.339799 17012 raft_consensus.cc:348] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:31.339900 17012 raft_consensus.cc:374] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:31.339932 17012 raft_consensus.cc:729] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:31.340044 17012 consensus_queue.cc:260] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:31.339905 17013 tablet_bootstrap.cc:492] T fdc5f76f63ec4528ad57202555caf243 P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:31.340305 17012 ts_tablet_manager.cc:1398] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:31.340507 17012 tablet_bootstrap.cc:492] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:31.341065 17013 tablet_bootstrap.cc:654] T fdc5f76f63ec4528ad57202555caf243 P 61b887f0d40d497badeca5604c1f0546: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:31.341219 17012 tablet_bootstrap.cc:654] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:31.341837 15605 tablet_service.cc:1467] Processing CreateTablet for tablet 604875eff3594876af54e9126057bd7f (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=1161fd279f6d435887536e9c124d251a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:31.342089 15605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 604875eff3594876af54e9126057bd7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:31.342103 17013 tablet_bootstrap.cc:492] T fdc5f76f63ec4528ad57202555caf243 P 61b887f0d40d497badeca5604c1f0546: No bootstrap required, opened a new log
I20240503 13:56:31.342170 17013 ts_tablet_manager.cc:1367] T fdc5f76f63ec4528ad57202555caf243 P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:56:31.342634 17013 raft_consensus.cc:348] T fdc5f76f63ec4528ad57202555caf243 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:31.342728 17013 raft_consensus.cc:374] T fdc5f76f63ec4528ad57202555caf243 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:31.342753 17013 raft_consensus.cc:729] T fdc5f76f63ec4528ad57202555caf243 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:31.342857 17013 consensus_queue.cc:260] T fdc5f76f63ec4528ad57202555caf243 P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:31.343118 17013 ts_tablet_manager.cc:1398] T fdc5f76f63ec4528ad57202555caf243 P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:31.343300 17013 tablet_bootstrap.cc:492] T 604875eff3594876af54e9126057bd7f P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:31.344166 17013 tablet_bootstrap.cc:654] T 604875eff3594876af54e9126057bd7f P 61b887f0d40d497badeca5604c1f0546: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:31.344439 15604 tablet_service.cc:1467] Processing CreateTablet for tablet fcb0aa9fce544ae2a6f1ca96c5315c36 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=1161fd279f6d435887536e9c124d251a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:31.344719 15604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcb0aa9fce544ae2a6f1ca96c5315c36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:31.345501 17012 tablet_bootstrap.cc:492] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:31.345592 17012 ts_tablet_manager.cc:1367] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:56:31.345662 17018 tablet_bootstrap.cc:492] T 604875eff3594876af54e9126057bd7f P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:31.346050 17012 raft_consensus.cc:348] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:31.346155 17012 raft_consensus.cc:374] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:31.346189 17012 raft_consensus.cc:729] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:31.346309 17012 consensus_queue.cc:260] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:31.346576 17012 ts_tablet_manager.cc:1398] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:31.346724 17012 tablet_bootstrap.cc:492] T fdc5f76f63ec4528ad57202555caf243 P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:31.346740 17018 tablet_bootstrap.cc:654] T 604875eff3594876af54e9126057bd7f P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:31.347254 16993 ts_tablet_manager.cc:1875] T 5e436c3a07b1401698c92cb0902c563a P e47105a0f46e4511b7112e571688a4c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:31.347596 17012 tablet_bootstrap.cc:654] T fdc5f76f63ec4528ad57202555caf243 P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:31.350309 16993 ts_tablet_manager.cc:1888] T 5e436c3a07b1401698c92cb0902c563a P e47105a0f46e4511b7112e571688a4c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:31.350592 17011 tablet_bootstrap.cc:492] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:31.350714 16993 log.cc:1196] T 5e436c3a07b1401698c92cb0902c563a P e47105a0f46e4511b7112e571688a4c7: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-7/wal/wals/5e436c3a07b1401698c92cb0902c563a
I20240503 13:56:31.351895 16993 ts_tablet_manager.cc:1909] T 5e436c3a07b1401698c92cb0902c563a P e47105a0f46e4511b7112e571688a4c7: Deleting consensus metadata
I20240503 13:56:31.352535 17012 tablet_bootstrap.cc:492] T fdc5f76f63ec4528ad57202555caf243 P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:31.352627 17012 ts_tablet_manager.cc:1367] T fdc5f76f63ec4528ad57202555caf243 P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 13:56:31.353092 17012 raft_consensus.cc:348] T fdc5f76f63ec4528ad57202555caf243 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:31.353226 17012 raft_consensus.cc:374] T fdc5f76f63ec4528ad57202555caf243 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:31.353263 17012 raft_consensus.cc:729] T fdc5f76f63ec4528ad57202555caf243 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:31.353395 17012 consensus_queue.cc:260] T fdc5f76f63ec4528ad57202555caf243 P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:31.353567 17011 tablet_bootstrap.cc:654] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:31.353667 17012 ts_tablet_manager.cc:1398] T fdc5f76f63ec4528ad57202555caf243 P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:31.354797 12896 catalog_manager.cc:4830] TS e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): tablet 5e436c3a07b1401698c92cb0902c563a (table rebalancer_dynamic_table_new_name [id=c2d29cf742b7499da66c50002c53a288]) successfully deleted
I20240503 13:56:31.354998 17011 tablet_bootstrap.cc:492] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:31.355106 17011 ts_tablet_manager.cc:1367] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:56:31.355293 17013 tablet_bootstrap.cc:492] T 604875eff3594876af54e9126057bd7f P 61b887f0d40d497badeca5604c1f0546: No bootstrap required, opened a new log
I20240503 13:56:31.355381 17013 ts_tablet_manager.cc:1367] T 604875eff3594876af54e9126057bd7f P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.000s
I20240503 13:56:31.355590 17011 raft_consensus.cc:348] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:31.355718 17011 raft_consensus.cc:374] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:31.355782 17011 raft_consensus.cc:729] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:31.355813 17013 raft_consensus.cc:348] T 604875eff3594876af54e9126057bd7f P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:31.355921 17013 raft_consensus.cc:374] T 604875eff3594876af54e9126057bd7f P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:31.355949 17013 raft_consensus.cc:729] T 604875eff3594876af54e9126057bd7f P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:31.355935 17011 consensus_queue.cc:260] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:31.356060 17013 consensus_queue.cc:260] T 604875eff3594876af54e9126057bd7f P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:31.356267 17011 ts_tablet_manager.cc:1398] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:31.356309 17013 ts_tablet_manager.cc:1398] T 604875eff3594876af54e9126057bd7f P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:31.358557 17018 tablet_bootstrap.cc:492] T 604875eff3594876af54e9126057bd7f P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:31.358659 17018 ts_tablet_manager.cc:1367] T 604875eff3594876af54e9126057bd7f P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.000s
I20240503 13:56:31.359609 17018 raft_consensus.cc:348] T 604875eff3594876af54e9126057bd7f P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:31.359788 17018 raft_consensus.cc:374] T 604875eff3594876af54e9126057bd7f P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:31.359885 17018 raft_consensus.cc:729] T 604875eff3594876af54e9126057bd7f P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:31.360138 17018 consensus_queue.cc:260] T 604875eff3594876af54e9126057bd7f P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:31.363821 17018 ts_tablet_manager.cc:1398] T 604875eff3594876af54e9126057bd7f P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:56:31.368791 17011 tablet_bootstrap.cc:492] T fdc5f76f63ec4528ad57202555caf243 P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:31.369902 17011 tablet_bootstrap.cc:654] T fdc5f76f63ec4528ad57202555caf243 P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:31.375639 17018 tablet_bootstrap.cc:492] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:31.377285 17011 tablet_bootstrap.cc:492] T fdc5f76f63ec4528ad57202555caf243 P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:31.378139 17011 ts_tablet_manager.cc:1367] T fdc5f76f63ec4528ad57202555caf243 P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 13:56:31.379468 17018 tablet_bootstrap.cc:654] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:31.379653 17011 raft_consensus.cc:348] T fdc5f76f63ec4528ad57202555caf243 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:31.380578 17011 raft_consensus.cc:374] T fdc5f76f63ec4528ad57202555caf243 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:31.380668 17011 raft_consensus.cc:729] T fdc5f76f63ec4528ad57202555caf243 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:31.380864 17011 consensus_queue.cc:260] T fdc5f76f63ec4528ad57202555caf243 P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:31.381182 17011 ts_tablet_manager.cc:1398] T fdc5f76f63ec4528ad57202555caf243 P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:31.382056 17018 tablet_bootstrap.cc:492] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:31.382206 17018 ts_tablet_manager.cc:1367] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20240503 13:56:31.382650 17018 raft_consensus.cc:348] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:31.382748 17018 raft_consensus.cc:374] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:31.382776 17018 raft_consensus.cc:729] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:31.382922 17018 consensus_queue.cc:260] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:31.383917 17018 ts_tablet_manager.cc:1398] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:31.396833 15604 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:56:31.398352 16887 raft_consensus.cc:482] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:31.398469 16887 raft_consensus.cc:504] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:31.398897 16887 leader_election.cc:288] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951), 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153)
I20240503 13:56:31.399327 15492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604875eff3594876af54e9126057bd7f" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546" is_pre_election: true
I20240503 13:56:31.399468 15492 raft_consensus.cc:2454] T 604875eff3594876af54e9126057bd7f P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 0.
I20240503 13:56:31.399443 15624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604875eff3594876af54e9126057bd7f" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d" is_pre_election: true
I20240503 13:56:31.399567 15624 raft_consensus.cc:2454] T 604875eff3594876af54e9126057bd7f P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 0.
I20240503 13:56:31.399824 15691 leader_election.cc:302] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 76c386bef61a4bd5a907266a5118ffe2; no voters:
I20240503 13:56:31.400621 16887 raft_consensus.cc:2781] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:31.400686 16887 raft_consensus.cc:482] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:31.400722 16887 raft_consensus.cc:3037] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:31.401873 16887 raft_consensus.cc:504] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:31.402257 16887 leader_election.cc:288] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 election: Requested vote from peers 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951), 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153)
I20240503 13:56:31.402464 15473 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:56:31.402649 15624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604875eff3594876af54e9126057bd7f" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d"
I20240503 13:56:31.402757 15624 raft_consensus.cc:3037] T 604875eff3594876af54e9126057bd7f P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:31.403611 15624 raft_consensus.cc:2454] T 604875eff3594876af54e9126057bd7f P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 1.
I20240503 13:56:31.403898 15691 leader_election.cc:302] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 76c386bef61a4bd5a907266a5118ffe2; no voters:
I20240503 13:56:31.404047 16887 raft_consensus.cc:2781] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:31.404116 16887 raft_consensus.cc:686] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 [term 1 LEADER]: Becoming Leader. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Running, Role: LEADER
I20240503 13:56:31.404245 16887 consensus_queue.cc:237] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:31.405838 13427 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 13:56:31.406505 12910 catalog_manager.cc:5484] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71). New cstate: current_term: 1 leader_uuid: "76c386bef61a4bd5a907266a5118ffe2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:31.407148 15493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604875eff3594876af54e9126057bd7f" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546"
I20240503 13:56:31.407258 15493 raft_consensus.cc:3037] T 604875eff3594876af54e9126057bd7f P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:31.408226 15493 raft_consensus.cc:2454] T 604875eff3594876af54e9126057bd7f P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 1.
I20240503 13:56:31.416662 13034 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:31.426319 15737 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:31.426846 12910 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:56:31.428550 15867 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:56:31.433266 15737 tablet_service.cc:1467] Processing CreateTablet for tablet 9a29eafcf3cb4fddbe3773a8aa6c3c38 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=484269f690b0450cba8d44725f06d26b]), partition=
I20240503 13:56:31.433266 15473 tablet_service.cc:1467] Processing CreateTablet for tablet 9a29eafcf3cb4fddbe3773a8aa6c3c38 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=484269f690b0450cba8d44725f06d26b]), partition=
I20240503 13:56:31.433532 15473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a29eafcf3cb4fddbe3773a8aa6c3c38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:31.433532 15737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a29eafcf3cb4fddbe3773a8aa6c3c38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:31.435928 17012 tablet_bootstrap.cc:492] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:31.436899 17012 tablet_bootstrap.cc:654] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:31.437446 17013 tablet_bootstrap.cc:492] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:31.438385 17013 tablet_bootstrap.cc:654] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P 61b887f0d40d497badeca5604c1f0546: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:31.440449 15867 tablet_service.cc:1467] Processing CreateTablet for tablet 9a29eafcf3cb4fddbe3773a8aa6c3c38 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=484269f690b0450cba8d44725f06d26b]), partition=
I20240503 13:56:31.440725 15867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a29eafcf3cb4fddbe3773a8aa6c3c38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:31.440979 12910 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalance_test_table_10" } modify_external_catalogs: true
I20240503 13:56:31.441120 12910 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalance_test_table_10" } modify_external_catalogs: true
I20240503 13:56:31.444860 12910 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Sending DeleteTablet for 3 replicas of tablet 67c4b85c17ab4e59be2d6223d993b48c
I20240503 13:56:31.445194 12910 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Sending DeleteTablet for 3 replicas of tablet 3d480dd03b6a4d9fb9f64c861d52a720
I20240503 13:56:31.445418 12910 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Sending DeleteTablet for 3 replicas of tablet ed3d2ea6e2d74bc5ba60dbbbf2e7b41d
I20240503 13:56:31.445475 13034 tablet_service.cc:1514] Processing DeleteTablet for tablet 67c4b85c17ab4e59be2d6223d993b48c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:31 UTC) from {username='slave'} at 127.0.0.1:59458
I20240503 13:56:31.445642 16085 tablet_replica.cc:330] T 67c4b85c17ab4e59be2d6223d993b48c P 60ba3ffe3b7746469834cd91c84be14e: stopping tablet replica
I20240503 13:56:31.445775 16085 raft_consensus.cc:2229] T 67c4b85c17ab4e59be2d6223d993b48c P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:31.445899 16085 raft_consensus.cc:2258] T 67c4b85c17ab4e59be2d6223d993b48c P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:31.446398 16085 ts_tablet_manager.cc:1875] T 67c4b85c17ab4e59be2d6223d993b48c P 60ba3ffe3b7746469834cd91c84be14e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:31.446399 13033 tablet_service.cc:1514] Processing DeleteTablet for tablet ed3d2ea6e2d74bc5ba60dbbbf2e7b41d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:31 UTC) from {username='slave'} at 127.0.0.1:59458
I20240503 13:56:31.448009 13295 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 13:56:31.448261 13164 tablet_service.cc:1514] Processing DeleteTablet for tablet 67c4b85c17ab4e59be2d6223d993b48c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:31 UTC) from {username='slave'} at 127.0.0.1:41668
I20240503 13:56:31.448506 13165 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 13:56:31.448740 13034 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d480dd03b6a4d9fb9f64c861d52a720 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:31 UTC) from {username='slave'} at 127.0.0.1:59458
I20240503 13:56:31.448792 16863 tablet_replica.cc:330] T 67c4b85c17ab4e59be2d6223d993b48c P 8ccf5a4ac450414c9d3579bd5800b770: stopping tablet replica
I20240503 13:56:31.448925 16863 raft_consensus.cc:2229] T 67c4b85c17ab4e59be2d6223d993b48c P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:31.449059 16863 raft_consensus.cc:2258] T 67c4b85c17ab4e59be2d6223d993b48c P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:31.449302 13295 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d480dd03b6a4d9fb9f64c861d52a720 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:31 UTC) from {username='slave'} at 127.0.0.1:33250
I20240503 13:56:31.449635 16863 ts_tablet_manager.cc:1875] T 67c4b85c17ab4e59be2d6223d993b48c P 8ccf5a4ac450414c9d3579bd5800b770: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:31.450773 16085 ts_tablet_manager.cc:1888] T 67c4b85c17ab4e59be2d6223d993b48c P 60ba3ffe3b7746469834cd91c84be14e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:56:31.450840 16085 log.cc:1196] T 67c4b85c17ab4e59be2d6223d993b48c P 60ba3ffe3b7746469834cd91c84be14e: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal/wals/67c4b85c17ab4e59be2d6223d993b48c
I20240503 13:56:31.451086 16085 ts_tablet_manager.cc:1909] T 67c4b85c17ab4e59be2d6223d993b48c P 60ba3ffe3b7746469834cd91c84be14e: Deleting consensus metadata
I20240503 13:56:31.452569 16085 tablet_replica.cc:330] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 60ba3ffe3b7746469834cd91c84be14e: stopping tablet replica
I20240503 13:56:31.452703 16085 raft_consensus.cc:2229] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:31.452708 12897 catalog_manager.cc:4830] TS 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695): tablet 67c4b85c17ab4e59be2d6223d993b48c (table rebalance_test_table_10 [id=0e569dec5058486fae2fb03b2b415c0f]) successfully deleted
I20240503 13:56:31.452780 16864 tablet_replica.cc:330] T 3d480dd03b6a4d9fb9f64c861d52a720 P ba0194098aa84449ae17e8a0d4c43703: stopping tablet replica
I20240503 13:56:31.452831 16085 raft_consensus.cc:2258] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:31.452912 16864 raft_consensus.cc:2229] T 3d480dd03b6a4d9fb9f64c861d52a720 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:31.453044 16864 raft_consensus.cc:2258] T 3d480dd03b6a4d9fb9f64c861d52a720 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:31.453047 13427 tablet_service.cc:1514] Processing DeleteTablet for tablet 67c4b85c17ab4e59be2d6223d993b48c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:31 UTC) from {username='slave'} at 127.0.0.1:33678
I20240503 13:56:31.453212 16087 tablet_replica.cc:330] T 67c4b85c17ab4e59be2d6223d993b48c P 7690ee79577c45e3a2c8f8b552c0bff8: stopping tablet replica
I20240503 13:56:31.453317 16085 ts_tablet_manager.cc:1875] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 60ba3ffe3b7746469834cd91c84be14e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:31.453307 13423 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d480dd03b6a4d9fb9f64c861d52a720 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:31 UTC) from {username='slave'} at 127.0.0.1:33678
I20240503 13:56:31.453464 13424 tablet_service.cc:1514] Processing DeleteTablet for tablet ed3d2ea6e2d74bc5ba60dbbbf2e7b41d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:31 UTC) from {username='slave'} at 127.0.0.1:33678
I20240503 13:56:31.453516 16864 ts_tablet_manager.cc:1875] T 3d480dd03b6a4d9fb9f64c861d52a720 P ba0194098aa84449ae17e8a0d4c43703: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:31.453321 16087 raft_consensus.cc:2229] T 67c4b85c17ab4e59be2d6223d993b48c P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:56:31.454165 16087 raft_consensus.cc:2258] T 67c4b85c17ab4e59be2d6223d993b48c P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:31.454666 16087 ts_tablet_manager.cc:1875] T 67c4b85c17ab4e59be2d6223d993b48c P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:31.455539 16864 ts_tablet_manager.cc:1888] T 3d480dd03b6a4d9fb9f64c861d52a720 P ba0194098aa84449ae17e8a0d4c43703: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:56:31.455600 16864 log.cc:1196] T 3d480dd03b6a4d9fb9f64c861d52a720 P ba0194098aa84449ae17e8a0d4c43703: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/wal/wals/3d480dd03b6a4d9fb9f64c861d52a720
I20240503 13:56:31.455827 16864 ts_tablet_manager.cc:1909] T 3d480dd03b6a4d9fb9f64c861d52a720 P ba0194098aa84449ae17e8a0d4c43703: Deleting consensus metadata
I20240503 13:56:31.456368 16085 ts_tablet_manager.cc:1888] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 60ba3ffe3b7746469834cd91c84be14e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:56:31.456430 16085 log.cc:1196] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 60ba3ffe3b7746469834cd91c84be14e: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal/wals/ed3d2ea6e2d74bc5ba60dbbbf2e7b41d
I20240503 13:56:31.456604 13164 tablet_service.cc:1514] Processing DeleteTablet for tablet ed3d2ea6e2d74bc5ba60dbbbf2e7b41d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:31 UTC) from {username='slave'} at 127.0.0.1:41668
I20240503 13:56:31.456679 16085 ts_tablet_manager.cc:1909] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 60ba3ffe3b7746469834cd91c84be14e: Deleting consensus metadata
I20240503 13:56:31.456972 12899 catalog_manager.cc:4830] TS ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279): tablet 3d480dd03b6a4d9fb9f64c861d52a720 (table rebalance_test_table_10 [id=0e569dec5058486fae2fb03b2b415c0f]) successfully deleted
I20240503 13:56:31.457440 16085 tablet_replica.cc:330] T 3d480dd03b6a4d9fb9f64c861d52a720 P 60ba3ffe3b7746469834cd91c84be14e: stopping tablet replica
I20240503 13:56:31.457561 16085 raft_consensus.cc:2229] T 3d480dd03b6a4d9fb9f64c861d52a720 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:31.457674 16085 raft_consensus.cc:2258] T 3d480dd03b6a4d9fb9f64c861d52a720 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:31.458272 16087 ts_tablet_manager.cc:1888] T 67c4b85c17ab4e59be2d6223d993b48c P 7690ee79577c45e3a2c8f8b552c0bff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:56:31.458340 16087 log.cc:1196] T 67c4b85c17ab4e59be2d6223d993b48c P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal/wals/67c4b85c17ab4e59be2d6223d993b48c
I20240503 13:56:31.458334 16085 ts_tablet_manager.cc:1875] T 3d480dd03b6a4d9fb9f64c861d52a720 P 60ba3ffe3b7746469834cd91c84be14e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:31.458583 16087 ts_tablet_manager.cc:1909] T 67c4b85c17ab4e59be2d6223d993b48c P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting consensus metadata
I20240503 13:56:31.459002 12897 catalog_manager.cc:4830] TS 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695): tablet ed3d2ea6e2d74bc5ba60dbbbf2e7b41d (table rebalance_test_table_10 [id=0e569dec5058486fae2fb03b2b415c0f]) successfully deleted
I20240503 13:56:31.460515 16863 ts_tablet_manager.cc:1888] T 67c4b85c17ab4e59be2d6223d993b48c P 8ccf5a4ac450414c9d3579bd5800b770: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:56:31.460598 16863 log.cc:1196] T 67c4b85c17ab4e59be2d6223d993b48c P 8ccf5a4ac450414c9d3579bd5800b770: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/wal/wals/67c4b85c17ab4e59be2d6223d993b48c
I20240503 13:56:31.460829 16863 ts_tablet_manager.cc:1909] T 67c4b85c17ab4e59be2d6223d993b48c P 8ccf5a4ac450414c9d3579bd5800b770: Deleting consensus metadata
I20240503 13:56:31.461427 12899 catalog_manager.cc:4830] TS 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387): tablet 67c4b85c17ab4e59be2d6223d993b48c (table rebalance_test_table_10 [id=0e569dec5058486fae2fb03b2b415c0f]) successfully deleted
I20240503 13:56:31.461480 16863 tablet_replica.cc:330] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 8ccf5a4ac450414c9d3579bd5800b770: stopping tablet replica
I20240503 13:56:31.461594 16863 raft_consensus.cc:2229] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:56:31.461766 16863 raft_consensus.cc:2258] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:31.462229 16863 ts_tablet_manager.cc:1875] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 8ccf5a4ac450414c9d3579bd5800b770: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:31.463924 16085 ts_tablet_manager.cc:1888] T 3d480dd03b6a4d9fb9f64c861d52a720 P 60ba3ffe3b7746469834cd91c84be14e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:56:31.464057 16085 log.cc:1196] T 3d480dd03b6a4d9fb9f64c861d52a720 P 60ba3ffe3b7746469834cd91c84be14e: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal/wals/3d480dd03b6a4d9fb9f64c861d52a720
I20240503 13:56:31.464339 16085 ts_tablet_manager.cc:1909] T 3d480dd03b6a4d9fb9f64c861d52a720 P 60ba3ffe3b7746469834cd91c84be14e: Deleting consensus metadata
I20240503 13:56:31.464380 16863 ts_tablet_manager.cc:1888] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 8ccf5a4ac450414c9d3579bd5800b770: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:56:31.464529 16863 log.cc:1196] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 8ccf5a4ac450414c9d3579bd5800b770: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/wal/wals/ed3d2ea6e2d74bc5ba60dbbbf2e7b41d
I20240503 13:56:31.464849 16863 ts_tablet_manager.cc:1909] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 8ccf5a4ac450414c9d3579bd5800b770: Deleting consensus metadata
I20240503 13:56:31.464884 17049 tablet_bootstrap.cc:492] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:31.465332 17013 tablet_bootstrap.cc:492] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P 61b887f0d40d497badeca5604c1f0546: No bootstrap required, opened a new log
I20240503 13:56:31.465452 17013 ts_tablet_manager.cc:1367] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.000s
I20240503 13:56:31.465734 12897 catalog_manager.cc:4830] TS 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695): tablet 3d480dd03b6a4d9fb9f64c861d52a720 (table rebalance_test_table_10 [id=0e569dec5058486fae2fb03b2b415c0f]) successfully deleted
I20240503 13:56:31.465747 12899 catalog_manager.cc:4830] TS 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387): tablet ed3d2ea6e2d74bc5ba60dbbbf2e7b41d (table rebalance_test_table_10 [id=0e569dec5058486fae2fb03b2b415c0f]) successfully deleted
I20240503 13:56:31.466115 17013 raft_consensus.cc:348] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:31.466229 17013 raft_consensus.cc:374] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:31.466264 17013 raft_consensus.cc:729] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:31.466387 17013 consensus_queue.cc:260] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:31.466652 17013 ts_tablet_manager.cc:1398] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:31.474905 17012 tablet_bootstrap.cc:492] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:31.475030 17012 ts_tablet_manager.cc:1367] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.039s user 0.002s sys 0.000s
I20240503 13:56:31.475049 17049 tablet_bootstrap.cc:654] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P e47105a0f46e4511b7112e571688a4c7: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:31.475469 17012 raft_consensus.cc:348] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:31.475587 17012 raft_consensus.cc:374] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:31.475620 17012 raft_consensus.cc:729] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:31.475744 17012 consensus_queue.cc:260] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:31.476016 17012 ts_tablet_manager.cc:1398] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:31.476883 17049 tablet_bootstrap.cc:492] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P e47105a0f46e4511b7112e571688a4c7: No bootstrap required, opened a new log
I20240503 13:56:31.477041 17049 ts_tablet_manager.cc:1367] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.001s
I20240503 13:56:31.477051 12896 catalog_manager.cc:4830] TS 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109): tablet 67c4b85c17ab4e59be2d6223d993b48c (table rebalance_test_table_10 [id=0e569dec5058486fae2fb03b2b415c0f]) successfully deleted
I20240503 13:56:31.477576 16087 tablet_replica.cc:330] T 3d480dd03b6a4d9fb9f64c861d52a720 P 7690ee79577c45e3a2c8f8b552c0bff8: stopping tablet replica
I20240503 13:56:31.477715 16087 raft_consensus.cc:2229] T 3d480dd03b6a4d9fb9f64c861d52a720 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:56:31.477911 16087 raft_consensus.cc:2258] T 3d480dd03b6a4d9fb9f64c861d52a720 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:31.478001 17049 raft_consensus.cc:348] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:31.478145 17049 raft_consensus.cc:374] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:31.478190 17049 raft_consensus.cc:729] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:31.478377 17049 consensus_queue.cc:260] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:31.478623 16119 raft_consensus.cc:482] T fdc5f76f63ec4528ad57202555caf243 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:31.478711 16119 raft_consensus.cc:504] T fdc5f76f63ec4528ad57202555caf243 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:31.479247 16119 leader_election.cc:288] T fdc5f76f63ec4528ad57202555caf243 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885), 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:31.480971 17049 ts_tablet_manager.cc:1398] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240503 13:56:31.482934 16087 ts_tablet_manager.cc:1875] T 3d480dd03b6a4d9fb9f64c861d52a720 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:31.485494 13052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc5f76f63ec4528ad57202555caf243" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e" is_pre_election: true
I20240503 13:56:31.485639 13052 raft_consensus.cc:2454] T fdc5f76f63ec4528ad57202555caf243 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 0.
I20240503 13:56:31.485934 15427 leader_election.cc:302] T fdc5f76f63ec4528ad57202555caf243 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 61b887f0d40d497badeca5604c1f0546; no voters:
I20240503 13:56:31.486102 16119 raft_consensus.cc:2781] T fdc5f76f63ec4528ad57202555caf243 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:31.486145 16119 raft_consensus.cc:482] T fdc5f76f63ec4528ad57202555caf243 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:31.486177 16119 raft_consensus.cc:3037] T fdc5f76f63ec4528ad57202555caf243 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:31.486186 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc5f76f63ec4528ad57202555caf243" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2" is_pre_election: true
I20240503 13:56:31.486301 15757 raft_consensus.cc:2454] T fdc5f76f63ec4528ad57202555caf243 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 0.
I20240503 13:56:31.487120 16119 raft_consensus.cc:504] T fdc5f76f63ec4528ad57202555caf243 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:31.487529 16119 leader_election.cc:288] T fdc5f76f63ec4528ad57202555caf243 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 election: Requested vote from peers 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885), 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:31.487821 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc5f76f63ec4528ad57202555caf243" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2"
I20240503 13:56:31.487938 15757 raft_consensus.cc:3037] T fdc5f76f63ec4528ad57202555caf243 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:31.487928 13052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc5f76f63ec4528ad57202555caf243" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e"
I20240503 13:56:31.488032 13052 raft_consensus.cc:3037] T fdc5f76f63ec4528ad57202555caf243 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:31.488854 15757 raft_consensus.cc:2454] T fdc5f76f63ec4528ad57202555caf243 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 1.
I20240503 13:56:31.489017 13052 raft_consensus.cc:2454] T fdc5f76f63ec4528ad57202555caf243 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 1.
I20240503 13:56:31.489180 15429 leader_election.cc:302] T fdc5f76f63ec4528ad57202555caf243 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61b887f0d40d497badeca5604c1f0546, 76c386bef61a4bd5a907266a5118ffe2; no voters:
I20240503 13:56:31.489516 16119 raft_consensus.cc:2781] T fdc5f76f63ec4528ad57202555caf243 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:31.489709 16119 raft_consensus.cc:686] T fdc5f76f63ec4528ad57202555caf243 P 61b887f0d40d497badeca5604c1f0546 [term 1 LEADER]: Becoming Leader. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Running, Role: LEADER
I20240503 13:56:31.489856 16119 consensus_queue.cc:237] T fdc5f76f63ec4528ad57202555caf243 P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:31.490768 16087 ts_tablet_manager.cc:1888] T 3d480dd03b6a4d9fb9f64c861d52a720 P 7690ee79577c45e3a2c8f8b552c0bff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:56:31.491101 16087 log.cc:1196] T 3d480dd03b6a4d9fb9f64c861d52a720 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal/wals/3d480dd03b6a4d9fb9f64c861d52a720
I20240503 13:56:31.491392 16087 ts_tablet_manager.cc:1909] T 3d480dd03b6a4d9fb9f64c861d52a720 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting consensus metadata
I20240503 13:56:31.491482 12910 catalog_manager.cc:5484] T fdc5f76f63ec4528ad57202555caf243 P 61b887f0d40d497badeca5604c1f0546 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61b887f0d40d497badeca5604c1f0546 (127.6.98.69). New cstate: current_term: 1 leader_uuid: "61b887f0d40d497badeca5604c1f0546" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } }
I20240503 13:56:31.492120 16087 tablet_replica.cc:330] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 7690ee79577c45e3a2c8f8b552c0bff8: stopping tablet replica
I20240503 13:56:31.492308 16087 raft_consensus.cc:2229] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:31.492209 12896 catalog_manager.cc:4830] TS 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109): tablet 3d480dd03b6a4d9fb9f64c861d52a720 (table rebalance_test_table_10 [id=0e569dec5058486fae2fb03b2b415c0f]) successfully deleted
I20240503 13:56:31.492426 16887 raft_consensus.cc:482] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:31.492547 16887 raft_consensus.cc:504] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:31.492935 16887 leader_election.cc:288] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951), 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:31.493104 16087 raft_consensus.cc:2258] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:31.493263 15624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb0aa9fce544ae2a6f1ca96c5315c36" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d" is_pre_election: true
I20240503 13:56:31.493389 15624 raft_consensus.cc:2454] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 0.
I20240503 13:56:31.493652 15691 leader_election.cc:302] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 76c386bef61a4bd5a907266a5118ffe2; no voters:
I20240503 13:56:31.493780 13052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb0aa9fce544ae2a6f1ca96c5315c36" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e" is_pre_election: true
I20240503 13:56:31.493832 16887 raft_consensus.cc:2781] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:31.493875 16887 raft_consensus.cc:482] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:31.493906 16887 raft_consensus.cc:3037] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:31.493908 13052 raft_consensus.cc:2454] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 0.
I20240503 13:56:31.494686 16087 ts_tablet_manager.cc:1875] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:31.494835 16887 raft_consensus.cc:504] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:31.495194 16887 leader_election.cc:288] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 election: Requested vote from peers 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951), 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:31.495524 15624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb0aa9fce544ae2a6f1ca96c5315c36" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d"
I20240503 13:56:31.495537 13052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb0aa9fce544ae2a6f1ca96c5315c36" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e"
I20240503 13:56:31.495635 13052 raft_consensus.cc:3037] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:31.495642 15624 raft_consensus.cc:3037] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:31.496338 13052 raft_consensus.cc:2454] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 1.
I20240503 13:56:31.496557 15624 raft_consensus.cc:2454] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 1.
I20240503 13:56:31.496657 15692 leader_election.cc:302] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 76c386bef61a4bd5a907266a5118ffe2; no voters:
I20240503 13:56:31.496946 16887 raft_consensus.cc:2781] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:31.497047 16887 raft_consensus.cc:686] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 LEADER]: Becoming Leader. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Running, Role: LEADER
I20240503 13:56:31.497176 16887 consensus_queue.cc:237] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:31.500391 16087 ts_tablet_manager.cc:1888] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 7690ee79577c45e3a2c8f8b552c0bff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:56:31.500478 16087 log.cc:1196] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal/wals/ed3d2ea6e2d74bc5ba60dbbbf2e7b41d
I20240503 13:56:31.504679 16087 ts_tablet_manager.cc:1909] T ed3d2ea6e2d74bc5ba60dbbbf2e7b41d P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting consensus metadata
I20240503 13:56:31.504803 12910 catalog_manager.cc:5484] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 76c386bef61a4bd5a907266a5118ffe2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71). New cstate: current_term: 1 leader_uuid: "76c386bef61a4bd5a907266a5118ffe2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } }
I20240503 13:56:31.505405 12896 catalog_manager.cc:4830] TS 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109): tablet ed3d2ea6e2d74bc5ba60dbbbf2e7b41d (table rebalance_test_table_10 [id=0e569dec5058486fae2fb03b2b415c0f]) successfully deleted
I20240503 13:56:31.514735 13316 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fd6a3d908840456f82c346c4bf1d8160"
dest_uuid: "ba0194098aa84449ae17e8a0d4c43703"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58230
I20240503 13:56:31.514921 13316 raft_consensus.cc:595] T fd6a3d908840456f82c346c4bf1d8160 P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Received request to transfer leadership
I20240503 13:56:31.684548 16119 raft_consensus.cc:482] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:31.684715 16119 raft_consensus.cc:504] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:31.685250 16119 leader_election.cc:288] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:31.685601 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a29eafcf3cb4fddbe3773a8aa6c3c38" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2" is_pre_election: true
I20240503 13:56:31.685753 15757 raft_consensus.cc:2454] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 0.
I20240503 13:56:31.686033 15429 leader_election.cc:302] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61b887f0d40d497badeca5604c1f0546, 76c386bef61a4bd5a907266a5118ffe2; no voters:
I20240503 13:56:31.686184 16119 raft_consensus.cc:2781] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:31.686249 16119 raft_consensus.cc:482] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:31.686302 16119 raft_consensus.cc:3037] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:31.687309 16119 raft_consensus.cc:504] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:31.687669 16119 leader_election.cc:288] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 election: Requested vote from peers 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:31.689100 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a29eafcf3cb4fddbe3773a8aa6c3c38" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2"
I20240503 13:56:31.689122 15888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a29eafcf3cb4fddbe3773a8aa6c3c38" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7" is_pre_election: true
I20240503 13:56:31.689213 15757 raft_consensus.cc:3037] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:31.689205 15889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a29eafcf3cb4fddbe3773a8aa6c3c38" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7"
I20240503 13:56:31.689245 15888 raft_consensus.cc:2454] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 0.
I20240503 13:56:31.690090 15757 raft_consensus.cc:2454] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 1.
I20240503 13:56:31.690402 15429 leader_election.cc:302] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61b887f0d40d497badeca5604c1f0546, 76c386bef61a4bd5a907266a5118ffe2; no voters: e47105a0f46e4511b7112e571688a4c7
I20240503 13:56:31.690661 16119 raft_consensus.cc:2781] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:31.690815 16119 raft_consensus.cc:686] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P 61b887f0d40d497badeca5604c1f0546 [term 1 LEADER]: Becoming Leader. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Running, Role: LEADER
I20240503 13:56:31.690981 16119 consensus_queue.cc:237] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:31.692723 12911 catalog_manager.cc:5484] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P 61b887f0d40d497badeca5604c1f0546 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61b887f0d40d497badeca5604c1f0546 (127.6.98.69). New cstate: current_term: 1 leader_uuid: "61b887f0d40d497badeca5604c1f0546" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:31.713897 15737 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:56:31.716049 15604 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:56:31.716548 13034 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:31.722683 13424 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:31.732198 13164 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 13:56:31.732885 13295 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 13:56:31.752388 15867 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:56:31.758369 15473 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:56:31.792464 16266 consensus_queue.cc:1035] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:31.795969 16887 consensus_queue.cc:1035] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:31.847826 12908 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:56:31.855895 15888 raft_consensus.cc:3037] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:31.856123 15757 raft_consensus.cc:1261] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Refusing update from remote peer 61b887f0d40d497badeca5604c1f0546: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:31.856986 17053 consensus_queue.cc:1035] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:31.857180 15888 raft_consensus.cc:1261] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Refusing update from remote peer 61b887f0d40d497badeca5604c1f0546: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:31.859344 17053 consensus_queue.cc:1035] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:31.860320 12906 catalog_manager.cc:5576] TS 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885) does not have the latest schema for tablet 9a29eafcf3cb4fddbe3773a8aa6c3c38 (table rebalancer_dynamic_table [id=484269f690b0450cba8d44725f06d26b]). Expected version 1 got 0
I20240503 13:56:31.863489 12906 catalog_manager.cc:5576] TS e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753) does not have the latest schema for tablet 9a29eafcf3cb4fddbe3773a8aa6c3c38 (table rebalancer_dynamic_table [id=484269f690b0450cba8d44725f06d26b]). Expected version 1 got 0
I20240503 13:56:31.866057 17168 tablet.cc:1722] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P e47105a0f46e4511b7112e571688a4c7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:31.866535 17168 tablet.cc:1620] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P e47105a0f46e4511b7112e571688a4c7: MemRowSet was empty: no flush needed.
I20240503 13:56:31.867877 17164 tablet.cc:1722] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P 61b887f0d40d497badeca5604c1f0546: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:31.868242 17164 tablet.cc:1620] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P 61b887f0d40d497badeca5604c1f0546: MemRowSet was empty: no flush needed.
I20240503 13:56:31.868400 12908 catalog_manager.cc:5576] TS 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885) does not have the latest schema for tablet 9a29eafcf3cb4fddbe3773a8aa6c3c38 (table rebalancer_dynamic_table [id=484269f690b0450cba8d44725f06d26b]). Expected version 1 got 0
I20240503 13:56:31.869246 12906 catalog_manager.cc:5576] TS e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753) does not have the latest schema for tablet 9a29eafcf3cb4fddbe3773a8aa6c3c38 (table rebalancer_dynamic_table [id=484269f690b0450cba8d44725f06d26b]). Expected version 1 got 0
I20240503 13:56:31.870246 17172 tablet.cc:1722] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P 76c386bef61a4bd5a907266a5118ffe2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:31.870810 17172 tablet.cc:1620] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P 76c386bef61a4bd5a907266a5118ffe2: MemRowSet was empty: no flush needed.
I20240503 13:56:31.875308 12905 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=484269f690b0450cba8d44725f06d26b] alter complete (version 1)
I20240503 13:56:31.881804 15604 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:56:31.882838 13034 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:31.887691 13424 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:31.892220 15867 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:56:31.900311 13295 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 13:56:31.900645 13164 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 13:56:31.913609 15473 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:56:31.929387 16890 raft_consensus.cc:979] T fd6a3d908840456f82c346c4bf1d8160 P ba0194098aa84449ae17e8a0d4c43703: : Instructing follower 60ba3ffe3b7746469834cd91c84be14e to start an election
I20240503 13:56:31.929503 16890 raft_consensus.cc:1067] T fd6a3d908840456f82c346c4bf1d8160 P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Signalling peer 60ba3ffe3b7746469834cd91c84be14e to start an election
I20240503 13:56:31.930142 16119 consensus_queue.cc:1035] T fdc5f76f63ec4528ad57202555caf243 P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:31.930785 13052 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fd6a3d908840456f82c346c4bf1d8160"
dest_uuid: "60ba3ffe3b7746469834cd91c84be14e"
from {username='slave'} at 127.6.98.67:46349
I20240503 13:56:31.930946 13052 raft_consensus.cc:482] T fd6a3d908840456f82c346c4bf1d8160 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 13:56:31.931006 13052 raft_consensus.cc:3037] T fd6a3d908840456f82c346c4bf1d8160 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:56:31.931962 13052 raft_consensus.cc:504] T fd6a3d908840456f82c346c4bf1d8160 P 60ba3ffe3b7746469834cd91c84be14e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:31.932703 13316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd6a3d908840456f82c346c4bf1d8160" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "ba0194098aa84449ae17e8a0d4c43703"
I20240503 13:56:31.932808 13316 raft_consensus.cc:3032] T fd6a3d908840456f82c346c4bf1d8160 P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 13:56:31.932844 13316 raft_consensus.cc:729] T fd6a3d908840456f82c346c4bf1d8160 P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Running, Role: LEADER
I20240503 13:56:31.933005 13316 consensus_queue.cc:260] T fd6a3d908840456f82c346c4bf1d8160 P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:31.933193 13316 raft_consensus.cc:3037] T fd6a3d908840456f82c346c4bf1d8160 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:56:31.934108 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd6a3d908840456f82c346c4bf1d8160" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
I20240503 13:56:31.934218 13316 raft_consensus.cc:2454] T fd6a3d908840456f82c346c4bf1d8160 P ba0194098aa84449ae17e8a0d4c43703 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 2.
I20240503 13:56:31.934235 13447 raft_consensus.cc:3037] T fd6a3d908840456f82c346c4bf1d8160 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:56:31.935321 13447 raft_consensus.cc:2454] T fd6a3d908840456f82c346c4bf1d8160 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 2.
I20240503 13:56:31.935673 12987 leader_election.cc:302] T fd6a3d908840456f82c346c4bf1d8160 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 7690ee79577c45e3a2c8f8b552c0bff8, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:31.935863 15958 raft_consensus.cc:2781] T fd6a3d908840456f82c346c4bf1d8160 P 60ba3ffe3b7746469834cd91c84be14e [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:56:31.935997 15958 raft_consensus.cc:686] T fd6a3d908840456f82c346c4bf1d8160 P 60ba3ffe3b7746469834cd91c84be14e [term 2 LEADER]: Becoming Leader. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Running, Role: LEADER
I20240503 13:56:31.936131 15958 consensus_queue.cc:237] T fd6a3d908840456f82c346c4bf1d8160 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:31.936236 13052 leader_election.cc:288] T fd6a3d908840456f82c346c4bf1d8160 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 2 election: Requested vote from peers ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279), 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153)
I20240503 13:56:31.938191 12906 catalog_manager.cc:5484] T fd6a3d908840456f82c346c4bf1d8160 P 60ba3ffe3b7746469834cd91c84be14e reported cstate change: term changed from 1 to 2, leader changed from ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) to 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65). New cstate: current_term: 2 leader_uuid: "60ba3ffe3b7746469834cd91c84be14e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:31.945163 15737 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:56:31.949335 17053 consensus_queue.cc:1035] T fdc5f76f63ec4528ad57202555caf243 P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:31.951661 15492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd6a3d908840456f82c346c4bf1d8160" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "61b887f0d40d497badeca5604c1f0546"
I20240503 13:56:31.951781 15492 raft_consensus.cc:3037] T fd6a3d908840456f82c346c4bf1d8160 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:56:31.952759 15492 raft_consensus.cc:2454] T fd6a3d908840456f82c346c4bf1d8160 P 61b887f0d40d497badeca5604c1f0546 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 2.
I20240503 13:56:31.953616 12908 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalance_test_table_12" } modify_external_catalogs: true
I20240503 13:56:31.953747 12908 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalance_test_table_12" } modify_external_catalogs: true
I20240503 13:56:31.956327 12908 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Sending DeleteTablet for 3 replicas of tablet 97dd13a5495a4ec7818d136e7cc95abd
I20240503 13:56:31.956655 12908 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Sending DeleteTablet for 3 replicas of tablet 2a7bc6049234402d806ce06fdb8df987
I20240503 13:56:31.956903 12908 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Sending DeleteTablet for 4 replicas of tablet 7a1c36d4fd0049139c98db7af595bc62
I20240503 13:56:31.956943 13034 tablet_service.cc:1514] Processing DeleteTablet for tablet 97dd13a5495a4ec7818d136e7cc95abd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:31 UTC) from {username='slave'} at 127.0.0.1:59458
I20240503 13:56:31.957111 16085 tablet_replica.cc:330] T 97dd13a5495a4ec7818d136e7cc95abd P 60ba3ffe3b7746469834cd91c84be14e: stopping tablet replica
I20240503 13:56:31.957216 13424 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a7bc6049234402d806ce06fdb8df987 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:31 UTC) from {username='slave'} at 127.0.0.1:33678
I20240503 13:56:31.957250 16085 raft_consensus.cc:2229] T 97dd13a5495a4ec7818d136e7cc95abd P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:31.957384 16087 tablet_replica.cc:330] T 2a7bc6049234402d806ce06fdb8df987 P 7690ee79577c45e3a2c8f8b552c0bff8: stopping tablet replica
I20240503 13:56:31.957369 13033 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a7bc6049234402d806ce06fdb8df987 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:31 UTC) from {username='slave'} at 127.0.0.1:59458
I20240503 13:56:31.957509 16087 raft_consensus.cc:2229] T 2a7bc6049234402d806ce06fdb8df987 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:56:31.957371 16085 raft_consensus.cc:2258] T 97dd13a5495a4ec7818d136e7cc95abd P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:31.957718 16087 raft_consensus.cc:2258] T 2a7bc6049234402d806ce06fdb8df987 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:31.957890 13033 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a1c36d4fd0049139c98db7af595bc62 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:31 UTC) from {username='slave'} at 127.0.0.1:59458
I20240503 13:56:31.957986 13295 tablet_service.cc:1514] Processing DeleteTablet for tablet 97dd13a5495a4ec7818d136e7cc95abd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:31 UTC) from {username='slave'} at 127.0.0.1:33250
I20240503 13:56:31.958146 16085 ts_tablet_manager.cc:1875] T 97dd13a5495a4ec7818d136e7cc95abd P 60ba3ffe3b7746469834cd91c84be14e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:31.958266 16087 ts_tablet_manager.cc:1875] T 2a7bc6049234402d806ce06fdb8df987 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:31.958281 13164 tablet_service.cc:1514] Processing DeleteTablet for tablet 97dd13a5495a4ec7818d136e7cc95abd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:31 UTC) from {username='slave'} at 127.0.0.1:41668
I20240503 13:56:31.958324 17178 tablet_replica.cc:330] T 97dd13a5495a4ec7818d136e7cc95abd P ba0194098aa84449ae17e8a0d4c43703: stopping tablet replica
I20240503 13:56:31.958423 13165 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a7bc6049234402d806ce06fdb8df987 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:31 UTC) from {username='slave'} at 127.0.0.1:41668
I20240503 13:56:31.958453 17178 raft_consensus.cc:2229] T 97dd13a5495a4ec7818d136e7cc95abd P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:56:31.958537 16863 tablet_replica.cc:330] T 97dd13a5495a4ec7818d136e7cc95abd P 8ccf5a4ac450414c9d3579bd5800b770: stopping tablet replica
I20240503 13:56:31.958652 16863 raft_consensus.cc:2229] T 97dd13a5495a4ec7818d136e7cc95abd P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:31.958755 16863 raft_consensus.cc:2258] T 97dd13a5495a4ec7818d136e7cc95abd P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:31.959209 16863 ts_tablet_manager.cc:1875] T 97dd13a5495a4ec7818d136e7cc95abd P 8ccf5a4ac450414c9d3579bd5800b770: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:31.959762 17178 raft_consensus.cc:2258] T 97dd13a5495a4ec7818d136e7cc95abd P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:31.960281 17178 ts_tablet_manager.cc:1875] T 97dd13a5495a4ec7818d136e7cc95abd P ba0194098aa84449ae17e8a0d4c43703: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:31.960326 16085 ts_tablet_manager.cc:1888] T 97dd13a5495a4ec7818d136e7cc95abd P 60ba3ffe3b7746469834cd91c84be14e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:56:31.960386 16085 log.cc:1196] T 97dd13a5495a4ec7818d136e7cc95abd P 60ba3ffe3b7746469834cd91c84be14e: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal/wals/97dd13a5495a4ec7818d136e7cc95abd
I20240503 13:56:31.960471 16087 ts_tablet_manager.cc:1888] T 2a7bc6049234402d806ce06fdb8df987 P 7690ee79577c45e3a2c8f8b552c0bff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:56:31.960556 16087 log.cc:1196] T 2a7bc6049234402d806ce06fdb8df987 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal/wals/2a7bc6049234402d806ce06fdb8df987
I20240503 13:56:31.960657 16085 ts_tablet_manager.cc:1909] T 97dd13a5495a4ec7818d136e7cc95abd P 60ba3ffe3b7746469834cd91c84be14e: Deleting consensus metadata
I20240503 13:56:31.960775 16087 ts_tablet_manager.cc:1909] T 2a7bc6049234402d806ce06fdb8df987 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting consensus metadata
I20240503 13:56:31.960845 13296 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a1c36d4fd0049139c98db7af595bc62 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:31 UTC) from {username='slave'} at 127.0.0.1:33250
I20240503 13:56:31.961374 16863 ts_tablet_manager.cc:1888] T 97dd13a5495a4ec7818d136e7cc95abd P 8ccf5a4ac450414c9d3579bd5800b770: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:56:31.961445 16863 log.cc:1196] T 97dd13a5495a4ec7818d136e7cc95abd P 8ccf5a4ac450414c9d3579bd5800b770: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/wal/wals/97dd13a5495a4ec7818d136e7cc95abd
I20240503 13:56:31.961491 16085 tablet_replica.cc:330] T 2a7bc6049234402d806ce06fdb8df987 P 60ba3ffe3b7746469834cd91c84be14e: stopping tablet replica
I20240503 13:56:31.961621 16085 raft_consensus.cc:2229] T 2a7bc6049234402d806ce06fdb8df987 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:31.961670 16863 ts_tablet_manager.cc:1909] T 97dd13a5495a4ec7818d136e7cc95abd P 8ccf5a4ac450414c9d3579bd5800b770: Deleting consensus metadata
I20240503 13:56:31.961659 15867 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a1c36d4fd0049139c98db7af595bc62 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:31 UTC) from {username='slave'} at 127.0.0.1:50498
I20240503 13:56:31.961719 16085 raft_consensus.cc:2258] T 2a7bc6049234402d806ce06fdb8df987 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:31.961710 12897 catalog_manager.cc:4830] TS 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695): tablet 97dd13a5495a4ec7818d136e7cc95abd (table rebalance_test_table_12 [id=3b862f5ca43e4df4815648fcc0bf383a]) successfully deleted
I20240503 13:56:31.962149 16085 ts_tablet_manager.cc:1875] T 2a7bc6049234402d806ce06fdb8df987 P 60ba3ffe3b7746469834cd91c84be14e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:31.962266 13424 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a1c36d4fd0049139c98db7af595bc62 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:31 UTC) from {username='slave'} at 127.0.0.1:33678
I20240503 13:56:31.962576 12899 catalog_manager.cc:4830] TS 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387): tablet 97dd13a5495a4ec7818d136e7cc95abd (table rebalance_test_table_12 [id=3b862f5ca43e4df4815648fcc0bf383a]) successfully deleted
I20240503 13:56:31.963047 17179 tablet_replica.cc:330] T 7a1c36d4fd0049139c98db7af595bc62 P e47105a0f46e4511b7112e571688a4c7: stopping tablet replica
I20240503 13:56:31.963205 17179 raft_consensus.cc:2229] T 7a1c36d4fd0049139c98db7af595bc62 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:31.963335 17179 raft_consensus.cc:2258] T 7a1c36d4fd0049139c98db7af595bc62 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:31.963340 12896 catalog_manager.cc:4830] TS 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109): tablet 2a7bc6049234402d806ce06fdb8df987 (table rebalance_test_table_12 [id=3b862f5ca43e4df4815648fcc0bf383a]) successfully deleted
I20240503 13:56:31.963454 16863 tablet_replica.cc:330] T 2a7bc6049234402d806ce06fdb8df987 P 8ccf5a4ac450414c9d3579bd5800b770: stopping tablet replica
I20240503 13:56:31.963586 16863 raft_consensus.cc:2229] T 2a7bc6049234402d806ce06fdb8df987 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:31.963594 16087 tablet_replica.cc:330] T 7a1c36d4fd0049139c98db7af595bc62 P 7690ee79577c45e3a2c8f8b552c0bff8: stopping tablet replica
I20240503 13:56:31.963730 16863 raft_consensus.cc:2258] T 2a7bc6049234402d806ce06fdb8df987 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:31.963744 16087 raft_consensus.cc:2229] T 7a1c36d4fd0049139c98db7af595bc62 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:31.963842 17179 ts_tablet_manager.cc:1875] T 7a1c36d4fd0049139c98db7af595bc62 P e47105a0f46e4511b7112e571688a4c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:31.963939 16087 raft_consensus.cc:2258] T 7a1c36d4fd0049139c98db7af595bc62 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:31.964080 16085 ts_tablet_manager.cc:1888] T 2a7bc6049234402d806ce06fdb8df987 P 60ba3ffe3b7746469834cd91c84be14e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:56:31.964146 16085 log.cc:1196] T 2a7bc6049234402d806ce06fdb8df987 P 60ba3ffe3b7746469834cd91c84be14e: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal/wals/2a7bc6049234402d806ce06fdb8df987
I20240503 13:56:31.964203 16863 ts_tablet_manager.cc:1875] T 2a7bc6049234402d806ce06fdb8df987 P 8ccf5a4ac450414c9d3579bd5800b770: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:31.964390 16085 ts_tablet_manager.cc:1909] T 2a7bc6049234402d806ce06fdb8df987 P 60ba3ffe3b7746469834cd91c84be14e: Deleting consensus metadata
I20240503 13:56:31.964689 16087 ts_tablet_manager.cc:1875] T 7a1c36d4fd0049139c98db7af595bc62 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:31.965739 12897 catalog_manager.cc:4830] TS 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695): tablet 2a7bc6049234402d806ce06fdb8df987 (table rebalance_test_table_12 [id=3b862f5ca43e4df4815648fcc0bf383a]) successfully deleted
I20240503 13:56:31.965798 16085 tablet_replica.cc:330] T 7a1c36d4fd0049139c98db7af595bc62 P 60ba3ffe3b7746469834cd91c84be14e: stopping tablet replica
I20240503 13:56:31.965935 16085 raft_consensus.cc:2229] T 7a1c36d4fd0049139c98db7af595bc62 P 60ba3ffe3b7746469834cd91c84be14e [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:56:31.966128 16085 raft_consensus.cc:2258] T 7a1c36d4fd0049139c98db7af595bc62 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:31.966303 16863 ts_tablet_manager.cc:1888] T 2a7bc6049234402d806ce06fdb8df987 P 8ccf5a4ac450414c9d3579bd5800b770: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:56:31.966477 16863 log.cc:1196] T 2a7bc6049234402d806ce06fdb8df987 P 8ccf5a4ac450414c9d3579bd5800b770: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/wal/wals/2a7bc6049234402d806ce06fdb8df987
I20240503 13:56:31.966557 16085 ts_tablet_manager.cc:1875] T 7a1c36d4fd0049139c98db7af595bc62 P 60ba3ffe3b7746469834cd91c84be14e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:31.966852 16863 ts_tablet_manager.cc:1909] T 2a7bc6049234402d806ce06fdb8df987 P 8ccf5a4ac450414c9d3579bd5800b770: Deleting consensus metadata
I20240503 13:56:31.967545 17179 ts_tablet_manager.cc:1888] T 7a1c36d4fd0049139c98db7af595bc62 P e47105a0f46e4511b7112e571688a4c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:31.967664 17179 log.cc:1196] T 7a1c36d4fd0049139c98db7af595bc62 P e47105a0f46e4511b7112e571688a4c7: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-7/wal/wals/7a1c36d4fd0049139c98db7af595bc62
I20240503 13:56:31.967816 16087 ts_tablet_manager.cc:1888] T 7a1c36d4fd0049139c98db7af595bc62 P 7690ee79577c45e3a2c8f8b552c0bff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:31.967943 16087 log.cc:1196] T 7a1c36d4fd0049139c98db7af595bc62 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal/wals/7a1c36d4fd0049139c98db7af595bc62
I20240503 13:56:31.967950 17179 ts_tablet_manager.cc:1909] T 7a1c36d4fd0049139c98db7af595bc62 P e47105a0f46e4511b7112e571688a4c7: Deleting consensus metadata
I20240503 13:56:31.968237 16087 ts_tablet_manager.cc:1909] T 7a1c36d4fd0049139c98db7af595bc62 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting consensus metadata
I20240503 13:56:31.968410 16085 ts_tablet_manager.cc:1888] T 7a1c36d4fd0049139c98db7af595bc62 P 60ba3ffe3b7746469834cd91c84be14e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:31.968677 12896 catalog_manager.cc:4830] TS e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): tablet 7a1c36d4fd0049139c98db7af595bc62 (table rebalance_test_table_12 [id=3b862f5ca43e4df4815648fcc0bf383a]) successfully deleted
I20240503 13:56:31.969220 12896 catalog_manager.cc:4830] TS 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109): tablet 7a1c36d4fd0049139c98db7af595bc62 (table rebalance_test_table_12 [id=3b862f5ca43e4df4815648fcc0bf383a]) successfully deleted
I20240503 13:56:31.970350 17178 ts_tablet_manager.cc:1888] T 97dd13a5495a4ec7818d136e7cc95abd P ba0194098aa84449ae17e8a0d4c43703: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:56:31.970463 16085 log.cc:1196] T 7a1c36d4fd0049139c98db7af595bc62 P 60ba3ffe3b7746469834cd91c84be14e: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal/wals/7a1c36d4fd0049139c98db7af595bc62
I20240503 13:56:31.970757 16085 ts_tablet_manager.cc:1909] T 7a1c36d4fd0049139c98db7af595bc62 P 60ba3ffe3b7746469834cd91c84be14e: Deleting consensus metadata
I20240503 13:56:31.970882 12899 catalog_manager.cc:4830] TS 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387): tablet 2a7bc6049234402d806ce06fdb8df987 (table rebalance_test_table_12 [id=3b862f5ca43e4df4815648fcc0bf383a]) successfully deleted
I20240503 13:56:31.971475 12897 catalog_manager.cc:4830] TS 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695): tablet 7a1c36d4fd0049139c98db7af595bc62 (table rebalance_test_table_12 [id=3b862f5ca43e4df4815648fcc0bf383a]) successfully deleted
I20240503 13:56:31.971645 17178 log.cc:1196] T 97dd13a5495a4ec7818d136e7cc95abd P ba0194098aa84449ae17e8a0d4c43703: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/wal/wals/97dd13a5495a4ec7818d136e7cc95abd
I20240503 13:56:31.971976 17178 ts_tablet_manager.cc:1909] T 97dd13a5495a4ec7818d136e7cc95abd P ba0194098aa84449ae17e8a0d4c43703: Deleting consensus metadata
I20240503 13:56:31.972656 12899 catalog_manager.cc:4830] TS ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279): tablet 97dd13a5495a4ec7818d136e7cc95abd (table rebalance_test_table_12 [id=3b862f5ca43e4df4815648fcc0bf383a]) successfully deleted
I20240503 13:56:31.972697 17178 tablet_replica.cc:330] T 7a1c36d4fd0049139c98db7af595bc62 P ba0194098aa84449ae17e8a0d4c43703: stopping tablet replica
I20240503 13:56:31.972910 17178 raft_consensus.cc:2229] T 7a1c36d4fd0049139c98db7af595bc62 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:31.973066 17178 raft_consensus.cc:2258] T 7a1c36d4fd0049139c98db7af595bc62 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:31.973582 17178 ts_tablet_manager.cc:1875] T 7a1c36d4fd0049139c98db7af595bc62 P ba0194098aa84449ae17e8a0d4c43703: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:31.975486 17178 ts_tablet_manager.cc:1888] T 7a1c36d4fd0049139c98db7af595bc62 P ba0194098aa84449ae17e8a0d4c43703: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:31.975610 17178 log.cc:1196] T 7a1c36d4fd0049139c98db7af595bc62 P ba0194098aa84449ae17e8a0d4c43703: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/wal/wals/7a1c36d4fd0049139c98db7af595bc62
I20240503 13:56:31.975934 17178 ts_tablet_manager.cc:1909] T 7a1c36d4fd0049139c98db7af595bc62 P ba0194098aa84449ae17e8a0d4c43703: Deleting consensus metadata
I20240503 13:56:31.976578 12899 catalog_manager.cc:4830] TS ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279): tablet 7a1c36d4fd0049139c98db7af595bc62 (table rebalance_test_table_12 [id=3b862f5ca43e4df4815648fcc0bf383a]) successfully deleted
I20240503 13:56:31.978314 16266 consensus_queue.cc:1035] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:31.981216 12908 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:56:31.985756 16266 consensus_queue.cc:1035] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:31.988303 17164 tablet.cc:1722] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P 61b887f0d40d497badeca5604c1f0546: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:31.988682 17164 tablet.cc:1620] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P 61b887f0d40d497badeca5604c1f0546: MemRowSet was empty: no flush needed.
I20240503 13:56:31.989672 17168 tablet.cc:1722] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P e47105a0f46e4511b7112e571688a4c7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:31.989991 17168 tablet.cc:1620] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P e47105a0f46e4511b7112e571688a4c7: MemRowSet was empty: no flush needed.
I20240503 13:56:31.991276 17172 tablet.cc:1722] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P 76c386bef61a4bd5a907266a5118ffe2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:31.991572 17172 tablet.cc:1620] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P 76c386bef61a4bd5a907266a5118ffe2: MemRowSet was empty: no flush needed.
I20240503 13:56:31.996023 12908 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=484269f690b0450cba8d44725f06d26b] alter complete (version 2)
I20240503 13:56:32.011410 13316 consensus_queue.cc:237] T 4647faf6a457415d8ae2b6691f452eab P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:32.012676 12905 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_extra_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:56:32.012804 13447 raft_consensus.cc:1261] T 4647faf6a457415d8ae2b6691f452eab P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:56:32.012831 15624 raft_consensus.cc:1261] T 4647faf6a457415d8ae2b6691f452eab P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:56:32.014305 16889 consensus_queue.cc:1035] T 4647faf6a457415d8ae2b6691f452eab P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:32.014518 16890 consensus_queue.cc:1035] T 4647faf6a457415d8ae2b6691f452eab P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:32.016790 16890 raft_consensus.cc:2932] T 4647faf6a457415d8ae2b6691f452eab P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:32.017397 13447 raft_consensus.cc:2932] T 4647faf6a457415d8ae2b6691f452eab P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:32.018635 15624 raft_consensus.cc:2932] T 4647faf6a457415d8ae2b6691f452eab P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:32.021791 15737 tablet_service.cc:1467] Processing CreateTablet for tablet 05e6080b2da14f8ca6f865820bbe77a4 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=e28614beb29c450da9e4f209b2fbe1b2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:32.021934 15472 tablet_service.cc:1467] Processing CreateTablet for tablet 27d5ced54d2e4aa9b6ea01fe8700443d (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=e28614beb29c450da9e4f209b2fbe1b2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:32.022095 15737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05e6080b2da14f8ca6f865820bbe77a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:32.022236 15472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27d5ced54d2e4aa9b6ea01fe8700443d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:32.022477 12906 catalog_manager.cc:5484] T 4647faf6a457415d8ae2b6691f452eab P 7690ee79577c45e3a2c8f8b552c0bff8 reported cstate change: config changed from index 4 to 5, VOTER 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66) evicted. New cstate: current_term: 1 leader_uuid: "ba0194098aa84449ae17e8a0d4c43703" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:32.021278 13296 tablet_service.cc:1467] Processing CreateTablet for tablet 27d5ced54d2e4aa9b6ea01fe8700443d (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=e28614beb29c450da9e4f209b2fbe1b2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:32.023900 13296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27d5ced54d2e4aa9b6ea01fe8700443d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:32.024868 13033 tablet_service.cc:1467] Processing CreateTablet for tablet 05e6080b2da14f8ca6f865820bbe77a4 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=e28614beb29c450da9e4f209b2fbe1b2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:32.024960 13295 tablet_service.cc:1467] Processing CreateTablet for tablet c8a53b7095e948f988c9151882bfa1bd (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=e28614beb29c450da9e4f209b2fbe1b2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:32.025131 13033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05e6080b2da14f8ca6f865820bbe77a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:32.025233 13295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8a53b7095e948f988c9151882bfa1bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:32.027850 17201 tablet_bootstrap.cc:492] T 05e6080b2da14f8ca6f865820bbe77a4 P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:32.027833 15473 tablet_service.cc:1467] Processing CreateTablet for tablet 05e6080b2da14f8ca6f865820bbe77a4 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=e28614beb29c450da9e4f209b2fbe1b2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:32.028108 15473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05e6080b2da14f8ca6f865820bbe77a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:32.028874 17202 tablet_bootstrap.cc:492] T 27d5ced54d2e4aa9b6ea01fe8700443d P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:32.028959 17201 tablet_bootstrap.cc:654] T 05e6080b2da14f8ca6f865820bbe77a4 P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:32.030185 17202 tablet_bootstrap.cc:654] T 27d5ced54d2e4aa9b6ea01fe8700443d P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:32.030306 17201 tablet_bootstrap.cc:492] T 05e6080b2da14f8ca6f865820bbe77a4 P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:32.030333 13165 tablet_service.cc:1514] Processing DeleteTablet for tablet 4647faf6a457415d8ae2b6691f452eab with delete_type TABLET_DATA_TOMBSTONED (TS 8ccf5a4ac450414c9d3579bd5800b770 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:41668
I20240503 13:56:32.030395 17201 ts_tablet_manager.cc:1367] T 05e6080b2da14f8ca6f865820bbe77a4 P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:56:32.030560 16863 tablet_replica.cc:330] T 4647faf6a457415d8ae2b6691f452eab P 8ccf5a4ac450414c9d3579bd5800b770: stopping tablet replica
I20240503 13:56:32.030705 16863 raft_consensus.cc:2229] T 4647faf6a457415d8ae2b6691f452eab P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:32.030828 16863 raft_consensus.cc:2258] T 4647faf6a457415d8ae2b6691f452eab P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:32.030880 17201 raft_consensus.cc:348] T 05e6080b2da14f8ca6f865820bbe77a4 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:32.031090 17201 raft_consensus.cc:374] T 05e6080b2da14f8ca6f865820bbe77a4 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:32.031126 17201 raft_consensus.cc:729] T 05e6080b2da14f8ca6f865820bbe77a4 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:32.031262 16863 ts_tablet_manager.cc:1875] T 4647faf6a457415d8ae2b6691f452eab P 8ccf5a4ac450414c9d3579bd5800b770: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:56:32.031246 17201 consensus_queue.cc:260] T 05e6080b2da14f8ca6f865820bbe77a4 P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:32.031538 17201 ts_tablet_manager.cc:1398] T 05e6080b2da14f8ca6f865820bbe77a4 P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:32.033428 16863 ts_tablet_manager.cc:1888] T 4647faf6a457415d8ae2b6691f452eab P 8ccf5a4ac450414c9d3579bd5800b770: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 13:56:32.033491 16863 log.cc:1196] T 4647faf6a457415d8ae2b6691f452eab P 8ccf5a4ac450414c9d3579bd5800b770: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/wal/wals/4647faf6a457415d8ae2b6691f452eab
I20240503 13:56:32.031980 17203 tablet_bootstrap.cc:492] T 05e6080b2da14f8ca6f865820bbe77a4 P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:32.034464 12899 catalog_manager.cc:4830] TS 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387): tablet 4647faf6a457415d8ae2b6691f452eab (table rebalance_test_table_3 [id=b600248fbc0b4c71b7c7cd8dd2844bbf]) successfully deleted
I20240503 13:56:32.034513 17203 tablet_bootstrap.cc:654] T 05e6080b2da14f8ca6f865820bbe77a4 P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:32.035056 15473 tablet_service.cc:1467] Processing CreateTablet for tablet c8a53b7095e948f988c9151882bfa1bd (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=e28614beb29c450da9e4f209b2fbe1b2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:32.035319 15473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8a53b7095e948f988c9151882bfa1bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:32.035768 15867 tablet_service.cc:1467] Processing CreateTablet for tablet 27d5ced54d2e4aa9b6ea01fe8700443d (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=e28614beb29c450da9e4f209b2fbe1b2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:32.035804 15604 tablet_service.cc:1467] Processing CreateTablet for tablet c8a53b7095e948f988c9151882bfa1bd (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=e28614beb29c450da9e4f209b2fbe1b2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:32.036036 15604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8a53b7095e948f988c9151882bfa1bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:32.036036 15867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27d5ced54d2e4aa9b6ea01fe8700443d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:32.039337 17204 tablet_bootstrap.cc:492] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:32.041752 17204 tablet_bootstrap.cc:654] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:32.042490 17207 tablet_bootstrap.cc:492] T c8a53b7095e948f988c9151882bfa1bd P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:32.042946 17204 tablet_bootstrap.cc:492] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546: No bootstrap required, opened a new log
I20240503 13:56:32.043287 17204 ts_tablet_manager.cc:1367] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:56:32.043879 17204 raft_consensus.cc:348] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:32.043998 17204 raft_consensus.cc:374] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:32.044021 17204 raft_consensus.cc:729] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:32.044153 17204 consensus_queue.cc:260] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:32.045150 17207 tablet_bootstrap.cc:654] T c8a53b7095e948f988c9151882bfa1bd P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:32.045183 17202 tablet_bootstrap.cc:492] T 27d5ced54d2e4aa9b6ea01fe8700443d P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:32.045284 17202 ts_tablet_manager.cc:1367] T 27d5ced54d2e4aa9b6ea01fe8700443d P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240503 13:56:32.045635 17202 raft_consensus.cc:348] T 27d5ced54d2e4aa9b6ea01fe8700443d P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:32.045727 17202 raft_consensus.cc:374] T 27d5ced54d2e4aa9b6ea01fe8700443d P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:32.045758 17202 raft_consensus.cc:729] T 27d5ced54d2e4aa9b6ea01fe8700443d P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:32.045872 17202 consensus_queue.cc:260] T 27d5ced54d2e4aa9b6ea01fe8700443d P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:32.046015 17204 ts_tablet_manager.cc:1398] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:56:32.046151 17202 ts_tablet_manager.cc:1398] T 27d5ced54d2e4aa9b6ea01fe8700443d P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:32.046180 17204 tablet_bootstrap.cc:492] T 05e6080b2da14f8ca6f865820bbe77a4 P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:32.046363 17202 tablet_bootstrap.cc:492] T c8a53b7095e948f988c9151882bfa1bd P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:32.047174 17202 tablet_bootstrap.cc:654] T c8a53b7095e948f988c9151882bfa1bd P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:32.047430 17203 tablet_bootstrap.cc:492] T 05e6080b2da14f8ca6f865820bbe77a4 P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:32.047561 17203 ts_tablet_manager.cc:1367] T 05e6080b2da14f8ca6f865820bbe77a4 P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240503 13:56:32.047983 17203 raft_consensus.cc:348] T 05e6080b2da14f8ca6f865820bbe77a4 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:32.048172 17203 raft_consensus.cc:374] T 05e6080b2da14f8ca6f865820bbe77a4 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:32.048218 17203 raft_consensus.cc:729] T 05e6080b2da14f8ca6f865820bbe77a4 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:32.048341 17203 consensus_queue.cc:260] T 05e6080b2da14f8ca6f865820bbe77a4 P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:32.048964 17208 tablet_bootstrap.cc:492] T 27d5ced54d2e4aa9b6ea01fe8700443d P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:32.049242 17203 ts_tablet_manager.cc:1398] T 05e6080b2da14f8ca6f865820bbe77a4 P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:32.050451 17208 tablet_bootstrap.cc:654] T 27d5ced54d2e4aa9b6ea01fe8700443d P e47105a0f46e4511b7112e571688a4c7: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:32.051070 17204 tablet_bootstrap.cc:654] T 05e6080b2da14f8ca6f865820bbe77a4 P 61b887f0d40d497badeca5604c1f0546: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:32.051704 17207 tablet_bootstrap.cc:492] T c8a53b7095e948f988c9151882bfa1bd P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:32.051805 17207 ts_tablet_manager.cc:1367] T c8a53b7095e948f988c9151882bfa1bd P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240503 13:56:32.051838 17208 tablet_bootstrap.cc:492] T 27d5ced54d2e4aa9b6ea01fe8700443d P e47105a0f46e4511b7112e571688a4c7: No bootstrap required, opened a new log
I20240503 13:56:32.051920 17208 ts_tablet_manager.cc:1367] T 27d5ced54d2e4aa9b6ea01fe8700443d P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:56:32.052354 17208 raft_consensus.cc:348] T 27d5ced54d2e4aa9b6ea01fe8700443d P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:32.052460 17208 raft_consensus.cc:374] T 27d5ced54d2e4aa9b6ea01fe8700443d P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:32.052528 17208 raft_consensus.cc:729] T 27d5ced54d2e4aa9b6ea01fe8700443d P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:32.052443 17207 raft_consensus.cc:348] T c8a53b7095e948f988c9151882bfa1bd P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:32.052599 17207 raft_consensus.cc:374] T c8a53b7095e948f988c9151882bfa1bd P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:32.052632 17207 raft_consensus.cc:729] T c8a53b7095e948f988c9151882bfa1bd P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:32.052659 17208 consensus_queue.cc:260] T 27d5ced54d2e4aa9b6ea01fe8700443d P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:32.052758 17207 consensus_queue.cc:260] T c8a53b7095e948f988c9151882bfa1bd P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:32.052934 17208 ts_tablet_manager.cc:1398] T 27d5ced54d2e4aa9b6ea01fe8700443d P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:32.053087 17207 ts_tablet_manager.cc:1398] T c8a53b7095e948f988c9151882bfa1bd P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:32.055477 17202 tablet_bootstrap.cc:492] T c8a53b7095e948f988c9151882bfa1bd P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:32.055591 17202 ts_tablet_manager.cc:1367] T c8a53b7095e948f988c9151882bfa1bd P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 13:56:32.056056 17202 raft_consensus.cc:348] T c8a53b7095e948f988c9151882bfa1bd P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:32.056174 17202 raft_consensus.cc:374] T c8a53b7095e948f988c9151882bfa1bd P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:32.056206 17202 raft_consensus.cc:729] T c8a53b7095e948f988c9151882bfa1bd P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:32.056330 17202 consensus_queue.cc:260] T c8a53b7095e948f988c9151882bfa1bd P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:32.056648 17202 ts_tablet_manager.cc:1398] T c8a53b7095e948f988c9151882bfa1bd P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:32.058959 17204 tablet_bootstrap.cc:492] T 05e6080b2da14f8ca6f865820bbe77a4 P 61b887f0d40d497badeca5604c1f0546: No bootstrap required, opened a new log
I20240503 13:56:32.059136 17204 ts_tablet_manager.cc:1367] T 05e6080b2da14f8ca6f865820bbe77a4 P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 13:56:32.059617 17204 raft_consensus.cc:348] T 05e6080b2da14f8ca6f865820bbe77a4 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:32.059762 17204 raft_consensus.cc:374] T 05e6080b2da14f8ca6f865820bbe77a4 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:32.059792 17204 raft_consensus.cc:729] T 05e6080b2da14f8ca6f865820bbe77a4 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:32.060184 17204 consensus_queue.cc:260] T 05e6080b2da14f8ca6f865820bbe77a4 P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:32.062186 17204 ts_tablet_manager.cc:1398] T 05e6080b2da14f8ca6f865820bbe77a4 P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:56:32.062841 17204 tablet_bootstrap.cc:492] T c8a53b7095e948f988c9151882bfa1bd P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:32.064018 17204 tablet_bootstrap.cc:654] T c8a53b7095e948f988c9151882bfa1bd P 61b887f0d40d497badeca5604c1f0546: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:32.069298 17204 tablet_bootstrap.cc:492] T c8a53b7095e948f988c9151882bfa1bd P 61b887f0d40d497badeca5604c1f0546: No bootstrap required, opened a new log
I20240503 13:56:32.069415 17204 ts_tablet_manager.cc:1367] T c8a53b7095e948f988c9151882bfa1bd P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 13:56:32.069916 17204 raft_consensus.cc:348] T c8a53b7095e948f988c9151882bfa1bd P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:32.070057 17204 raft_consensus.cc:374] T c8a53b7095e948f988c9151882bfa1bd P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:32.070122 17204 raft_consensus.cc:729] T c8a53b7095e948f988c9151882bfa1bd P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:32.070279 17204 consensus_queue.cc:260] T c8a53b7095e948f988c9151882bfa1bd P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:32.070619 17204 ts_tablet_manager.cc:1398] T c8a53b7095e948f988c9151882bfa1bd P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:32.092361 15958 raft_consensus.cc:482] T 05e6080b2da14f8ca6f865820bbe77a4 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:32.092571 15958 raft_consensus.cc:504] T 05e6080b2da14f8ca6f865820bbe77a4 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:32.092957 15958 leader_election.cc:288] T 05e6080b2da14f8ca6f865820bbe77a4 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885), 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153)
I20240503 13:56:32.093454 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e6080b2da14f8ca6f865820bbe77a4" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2" is_pre_election: true
I20240503 13:56:32.093497 15492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e6080b2da14f8ca6f865820bbe77a4" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546" is_pre_election: true
I20240503 13:56:32.093609 15492 raft_consensus.cc:2454] T 05e6080b2da14f8ca6f865820bbe77a4 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 0.
I20240503 13:56:32.093609 15757 raft_consensus.cc:2454] T 05e6080b2da14f8ca6f865820bbe77a4 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 0.
I20240503 13:56:32.093859 12990 leader_election.cc:302] T 05e6080b2da14f8ca6f865820bbe77a4 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 61b887f0d40d497badeca5604c1f0546; no voters:
I20240503 13:56:32.094048 15958 raft_consensus.cc:2781] T 05e6080b2da14f8ca6f865820bbe77a4 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:32.094105 15958 raft_consensus.cc:482] T 05e6080b2da14f8ca6f865820bbe77a4 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:32.094141 15958 raft_consensus.cc:3037] T 05e6080b2da14f8ca6f865820bbe77a4 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:32.095301 15958 raft_consensus.cc:504] T 05e6080b2da14f8ca6f865820bbe77a4 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:32.095676 15958 leader_election.cc:288] T 05e6080b2da14f8ca6f865820bbe77a4 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 election: Requested vote from peers 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885), 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153)
I20240503 13:56:32.096036 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e6080b2da14f8ca6f865820bbe77a4" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2"
I20240503 13:56:32.096206 15757 raft_consensus.cc:3037] T 05e6080b2da14f8ca6f865820bbe77a4 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:32.097477 15757 raft_consensus.cc:2454] T 05e6080b2da14f8ca6f865820bbe77a4 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 1.
I20240503 13:56:32.097764 15492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e6080b2da14f8ca6f865820bbe77a4" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546"
I20240503 13:56:32.097798 12990 leader_election.cc:302] T 05e6080b2da14f8ca6f865820bbe77a4 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 76c386bef61a4bd5a907266a5118ffe2; no voters:
I20240503 13:56:32.097882 15492 raft_consensus.cc:3037] T 05e6080b2da14f8ca6f865820bbe77a4 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:32.098017 15958 raft_consensus.cc:2781] T 05e6080b2da14f8ca6f865820bbe77a4 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:32.098850 15492 raft_consensus.cc:2454] T 05e6080b2da14f8ca6f865820bbe77a4 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 1.
I20240503 13:56:32.100369 15958 raft_consensus.cc:686] T 05e6080b2da14f8ca6f865820bbe77a4 P 60ba3ffe3b7746469834cd91c84be14e [term 1 LEADER]: Becoming Leader. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Running, Role: LEADER
I20240503 13:56:32.100842 15958 consensus_queue.cc:237] T 05e6080b2da14f8ca6f865820bbe77a4 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:32.100876 12910 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:56:32.104790 12910 catalog_manager.cc:5484] T 05e6080b2da14f8ca6f865820bbe77a4 P 60ba3ffe3b7746469834cd91c84be14e reported cstate change: term changed from 0 to 1, leader changed from <none> to 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65). New cstate: current_term: 1 leader_uuid: "60ba3ffe3b7746469834cd91c84be14e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:32.109858 17164 tablet.cc:1722] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P 61b887f0d40d497badeca5604c1f0546: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:32.110311 17168 tablet.cc:1722] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P e47105a0f46e4511b7112e571688a4c7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:32.110749 17172 tablet.cc:1722] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P 76c386bef61a4bd5a907266a5118ffe2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:32.119159 12904 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table_new_name [id=484269f690b0450cba8d44725f06d26b] alter complete (version 3)
I20240503 13:56:32.120554 17258 raft_consensus.cc:482] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:32.120679 17258 raft_consensus.cc:504] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:32.121758 15888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27d5ced54d2e4aa9b6ea01fe8700443d" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7" is_pre_election: true
I20240503 13:56:32.121922 15888 raft_consensus.cc:2454] T 27d5ced54d2e4aa9b6ea01fe8700443d P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 0.
I20240503 13:56:32.122351 15426 leader_election.cc:302] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61b887f0d40d497badeca5604c1f0546, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:32.122669 17053 raft_consensus.cc:2781] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:32.122735 17053 raft_consensus.cc:482] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:32.122779 17053 raft_consensus.cc:3037] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:32.123896 17053 raft_consensus.cc:504] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:32.124233 17053 leader_election.cc:288] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 election: Requested vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:32.124922 13316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27d5ced54d2e4aa9b6ea01fe8700443d" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703"
I20240503 13:56:32.125041 13316 raft_consensus.cc:3037] T 27d5ced54d2e4aa9b6ea01fe8700443d P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:32.126057 13316 raft_consensus.cc:2454] T 27d5ced54d2e4aa9b6ea01fe8700443d P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 1.
I20240503 13:56:32.126389 15429 leader_election.cc:302] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61b887f0d40d497badeca5604c1f0546, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:32.126569 17053 raft_consensus.cc:2781] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:32.126636 17053 raft_consensus.cc:686] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 [term 1 LEADER]: Becoming Leader. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Running, Role: LEADER
I20240503 13:56:32.126766 17053 consensus_queue.cc:237] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:32.127089 15888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27d5ced54d2e4aa9b6ea01fe8700443d" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7"
I20240503 13:56:32.127218 15888 raft_consensus.cc:3037] T 27d5ced54d2e4aa9b6ea01fe8700443d P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:32.128192 15888 raft_consensus.cc:2454] T 27d5ced54d2e4aa9b6ea01fe8700443d P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 1.
I20240503 13:56:32.129050 12903 catalog_manager.cc:5484] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61b887f0d40d497badeca5604c1f0546 (127.6.98.69). New cstate: current_term: 1 leader_uuid: "61b887f0d40d497badeca5604c1f0546" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:32.130311 13316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27d5ced54d2e4aa9b6ea01fe8700443d" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703" is_pre_election: true
I20240503 13:56:32.130434 13316 raft_consensus.cc:2362] T 27d5ced54d2e4aa9b6ea01fe8700443d P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 1. Re-sending same reply.
I20240503 13:56:32.130795 17258 leader_election.cc:288] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:32.217908 16248 raft_consensus.cc:482] T c8a53b7095e948f988c9151882bfa1bd P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:32.218039 16248 raft_consensus.cc:504] T c8a53b7095e948f988c9151882bfa1bd P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:32.218485 16248 leader_election.cc:288] T c8a53b7095e948f988c9151882bfa1bd P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279), 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153)
I20240503 13:56:32.218814 13316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8a53b7095e948f988c9151882bfa1bd" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703" is_pre_election: true
I20240503 13:56:32.218966 13316 raft_consensus.cc:2454] T c8a53b7095e948f988c9151882bfa1bd P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 0.
I20240503 13:56:32.219697 15561 leader_election.cc:302] T c8a53b7095e948f988c9151882bfa1bd P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:32.219941 16248 raft_consensus.cc:2781] T c8a53b7095e948f988c9151882bfa1bd P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:32.219997 16248 raft_consensus.cc:482] T c8a53b7095e948f988c9151882bfa1bd P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:32.220029 16248 raft_consensus.cc:3037] T c8a53b7095e948f988c9151882bfa1bd P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:32.220989 16248 raft_consensus.cc:504] T c8a53b7095e948f988c9151882bfa1bd P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:32.221305 16248 leader_election.cc:288] T c8a53b7095e948f988c9151882bfa1bd P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 election: Requested vote from peers ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279), 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153)
I20240503 13:56:32.221657 13316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8a53b7095e948f988c9151882bfa1bd" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703"
I20240503 13:56:32.221801 13316 raft_consensus.cc:3037] T c8a53b7095e948f988c9151882bfa1bd P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:32.222718 13316 raft_consensus.cc:2454] T c8a53b7095e948f988c9151882bfa1bd P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 1.
I20240503 13:56:32.222954 12905 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:32.223091 12905 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:32.223235 15561 leader_election.cc:302] T c8a53b7095e948f988c9151882bfa1bd P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:32.223539 16248 raft_consensus.cc:2781] T c8a53b7095e948f988c9151882bfa1bd P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:32.223731 16248 raft_consensus.cc:686] T c8a53b7095e948f988c9151882bfa1bd P 009f849ee7b54c80a3f5264695047e4d [term 1 LEADER]: Becoming Leader. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Running, Role: LEADER
I20240503 13:56:32.223747 15492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8a53b7095e948f988c9151882bfa1bd" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546" is_pre_election: true
I20240503 13:56:32.223879 15492 raft_consensus.cc:2454] T c8a53b7095e948f988c9151882bfa1bd P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 0.
I20240503 13:56:32.223872 16248 consensus_queue.cc:237] T c8a53b7095e948f988c9151882bfa1bd P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:32.224062 15493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8a53b7095e948f988c9151882bfa1bd" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546"
I20240503 13:56:32.224148 15493 raft_consensus.cc:3037] T c8a53b7095e948f988c9151882bfa1bd P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:32.225040 15493 raft_consensus.cc:2454] T c8a53b7095e948f988c9151882bfa1bd P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 1.
I20240503 13:56:32.226469 12905 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Sending DeleteTablet for 3 replicas of tablet 9a29eafcf3cb4fddbe3773a8aa6c3c38
I20240503 13:56:32.226805 12904 catalog_manager.cc:5484] T c8a53b7095e948f988c9151882bfa1bd P 009f849ee7b54c80a3f5264695047e4d reported cstate change: term changed from 0 to 1, leader changed from <none> to 009f849ee7b54c80a3f5264695047e4d (127.6.98.70). New cstate: current_term: 1 leader_uuid: "009f849ee7b54c80a3f5264695047e4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:32.227190 15473 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a29eafcf3cb4fddbe3773a8aa6c3c38 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:32 UTC) from {username='slave'} at 127.0.0.1:58758
I20240503 13:56:32.227564 15737 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a29eafcf3cb4fddbe3773a8aa6c3c38 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:32 UTC) from {username='slave'} at 127.0.0.1:44984
I20240503 13:56:32.227793 15867 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a29eafcf3cb4fddbe3773a8aa6c3c38 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:32 UTC) from {username='slave'} at 127.0.0.1:50498
I20240503 13:56:32.228003 17179 tablet_replica.cc:330] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P e47105a0f46e4511b7112e571688a4c7: stopping tablet replica
I20240503 13:56:32.228138 17179 raft_consensus.cc:2229] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:32.228261 17179 raft_consensus.cc:2258] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:32.228886 17263 tablet_replica.cc:330] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P 61b887f0d40d497badeca5604c1f0546: stopping tablet replica
I20240503 13:56:32.228978 17264 tablet_replica.cc:330] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P 76c386bef61a4bd5a907266a5118ffe2: stopping tablet replica
I20240503 13:56:32.229072 17263 raft_consensus.cc:2229] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P 61b887f0d40d497badeca5604c1f0546 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:56:32.229085 17179 ts_tablet_manager.cc:1875] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P e47105a0f46e4511b7112e571688a4c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:32.229148 17264 raft_consensus.cc:2229] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:32.229277 17264 raft_consensus.cc:2258] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:32.229489 17263 raft_consensus.cc:2258] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:32.230131 17263 ts_tablet_manager.cc:1875] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P 61b887f0d40d497badeca5604c1f0546: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:32.230794 17264 ts_tablet_manager.cc:1875] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P 76c386bef61a4bd5a907266a5118ffe2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:32.231221 17179 ts_tablet_manager.cc:1888] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P e47105a0f46e4511b7112e571688a4c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:32.231292 17179 log.cc:1196] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P e47105a0f46e4511b7112e571688a4c7: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-7/wal/wals/9a29eafcf3cb4fddbe3773a8aa6c3c38
I20240503 13:56:32.231619 17179 ts_tablet_manager.cc:1909] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P e47105a0f46e4511b7112e571688a4c7: Deleting consensus metadata
I20240503 13:56:32.232468 17263 ts_tablet_manager.cc:1888] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P 61b887f0d40d497badeca5604c1f0546: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:32.232582 17263 log.cc:1196] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P 61b887f0d40d497badeca5604c1f0546: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/wal/wals/9a29eafcf3cb4fddbe3773a8aa6c3c38
I20240503 13:56:32.232800 17264 ts_tablet_manager.cc:1888] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P 76c386bef61a4bd5a907266a5118ffe2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:32.232895 17264 log.cc:1196] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P 76c386bef61a4bd5a907266a5118ffe2: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/wal/wals/9a29eafcf3cb4fddbe3773a8aa6c3c38
I20240503 13:56:32.233180 17264 ts_tablet_manager.cc:1909] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P 76c386bef61a4bd5a907266a5118ffe2: Deleting consensus metadata
I20240503 13:56:32.233196 17263 ts_tablet_manager.cc:1909] T 9a29eafcf3cb4fddbe3773a8aa6c3c38 P 61b887f0d40d497badeca5604c1f0546: Deleting consensus metadata
I20240503 13:56:32.233914 12899 catalog_manager.cc:4830] TS 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885): tablet 9a29eafcf3cb4fddbe3773a8aa6c3c38 (table rebalancer_dynamic_table_new_name [id=484269f690b0450cba8d44725f06d26b]) successfully deleted
I20240503 13:56:32.234090 12899 catalog_manager.cc:4830] TS 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153): tablet 9a29eafcf3cb4fddbe3773a8aa6c3c38 (table rebalancer_dynamic_table_new_name [id=484269f690b0450cba8d44725f06d26b]) successfully deleted
I20240503 13:56:32.234509 12896 catalog_manager.cc:4830] TS e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): tablet 9a29eafcf3cb4fddbe3773a8aa6c3c38 (table rebalancer_dynamic_table_new_name [id=484269f690b0450cba8d44725f06d26b]) successfully deleted
I20240503 13:56:32.246721 15473 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:56:32.248600 15604 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:32.248628 13424 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:32.249708 15737 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:56:32.254869 13033 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:32.262768 13165 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 13:56:32.272090 15867 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:56:32.276748 13295 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 13:56:32.327762 13447 consensus_queue.cc:237] T b291e3dd6a7a454ca4d736eff8dd3a41 P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:32.329226 13183 raft_consensus.cc:1261] T b291e3dd6a7a454ca4d736eff8dd3a41 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Refusing update from remote peer 7690ee79577c45e3a2c8f8b552c0bff8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:56:32.329282 15493 raft_consensus.cc:1261] T b291e3dd6a7a454ca4d736eff8dd3a41 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Refusing update from remote peer 7690ee79577c45e3a2c8f8b552c0bff8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:56:32.329643 12904 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:56:32.329700 17056 consensus_queue.cc:1035] T b291e3dd6a7a454ca4d736eff8dd3a41 P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:32.330127 16268 consensus_queue.cc:1035] T b291e3dd6a7a454ca4d736eff8dd3a41 P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:32.332152 17056 raft_consensus.cc:2932] T b291e3dd6a7a454ca4d736eff8dd3a41 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:32.332821 15493 raft_consensus.cc:2932] T b291e3dd6a7a454ca4d736eff8dd3a41 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:32.335441 12904 catalog_manager.cc:5484] T b291e3dd6a7a454ca4d736eff8dd3a41 P 7690ee79577c45e3a2c8f8b552c0bff8 reported cstate change: config changed from index 4 to 5, VOTER ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) evicted. New cstate: current_term: 1 leader_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:32.335973 15604 tablet_service.cc:1467] Processing CreateTablet for tablet fd2db190ece74343b3e11b3de00c17cc (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2026ccafb8d246a4ac6bca16a95bb464]), partition=
I20240503 13:56:32.336210 15604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd2db190ece74343b3e11b3de00c17cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:32.336418 15473 tablet_service.cc:1467] Processing CreateTablet for tablet fd2db190ece74343b3e11b3de00c17cc (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2026ccafb8d246a4ac6bca16a95bb464]), partition=
I20240503 13:56:32.336671 15473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd2db190ece74343b3e11b3de00c17cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:32.336671 15737 tablet_service.cc:1467] Processing CreateTablet for tablet fd2db190ece74343b3e11b3de00c17cc (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2026ccafb8d246a4ac6bca16a95bb464]), partition=
I20240503 13:56:32.336889 15737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd2db190ece74343b3e11b3de00c17cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:32.338321 17204 tablet_bootstrap.cc:492] T fd2db190ece74343b3e11b3de00c17cc P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:32.338981 17207 tablet_bootstrap.cc:492] T fd2db190ece74343b3e11b3de00c17cc P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:32.339224 17204 tablet_bootstrap.cc:654] T fd2db190ece74343b3e11b3de00c17cc P 61b887f0d40d497badeca5604c1f0546: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:32.339831 17207 tablet_bootstrap.cc:654] T fd2db190ece74343b3e11b3de00c17cc P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:32.339972 13183 raft_consensus.cc:2932] T b291e3dd6a7a454ca4d736eff8dd3a41 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:32.340440 17201 tablet_bootstrap.cc:492] T fd2db190ece74343b3e11b3de00c17cc P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:32.340847 17207 tablet_bootstrap.cc:492] T fd2db190ece74343b3e11b3de00c17cc P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:32.340909 17207 ts_tablet_manager.cc:1367] T fd2db190ece74343b3e11b3de00c17cc P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:56:32.341225 17201 tablet_bootstrap.cc:654] T fd2db190ece74343b3e11b3de00c17cc P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:32.341369 17207 raft_consensus.cc:348] T fd2db190ece74343b3e11b3de00c17cc P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:32.341451 17207 raft_consensus.cc:374] T fd2db190ece74343b3e11b3de00c17cc P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:32.341471 17207 raft_consensus.cc:729] T fd2db190ece74343b3e11b3de00c17cc P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:32.341580 17207 consensus_queue.cc:260] T fd2db190ece74343b3e11b3de00c17cc P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:32.341836 17207 ts_tablet_manager.cc:1398] T fd2db190ece74343b3e11b3de00c17cc P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:32.342257 17201 tablet_bootstrap.cc:492] T fd2db190ece74343b3e11b3de00c17cc P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:32.342368 17201 ts_tablet_manager.cc:1367] T fd2db190ece74343b3e11b3de00c17cc P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:56:32.342837 17201 raft_consensus.cc:348] T fd2db190ece74343b3e11b3de00c17cc P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:32.342939 17201 raft_consensus.cc:374] T fd2db190ece74343b3e11b3de00c17cc P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:32.342967 17201 raft_consensus.cc:729] T fd2db190ece74343b3e11b3de00c17cc P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:32.343102 17201 consensus_queue.cc:260] T fd2db190ece74343b3e11b3de00c17cc P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:32.343372 17201 ts_tablet_manager.cc:1398] T fd2db190ece74343b3e11b3de00c17cc P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:32.344918 17204 tablet_bootstrap.cc:492] T fd2db190ece74343b3e11b3de00c17cc P 61b887f0d40d497badeca5604c1f0546: No bootstrap required, opened a new log
I20240503 13:56:32.345022 17204 ts_tablet_manager.cc:1367] T fd2db190ece74343b3e11b3de00c17cc P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 13:56:32.345489 17204 raft_consensus.cc:348] T fd2db190ece74343b3e11b3de00c17cc P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:32.345595 17204 raft_consensus.cc:374] T fd2db190ece74343b3e11b3de00c17cc P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:32.345623 17204 raft_consensus.cc:729] T fd2db190ece74343b3e11b3de00c17cc P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:32.345736 17204 consensus_queue.cc:260] T fd2db190ece74343b3e11b3de00c17cc P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:32.345996 17204 ts_tablet_manager.cc:1398] T fd2db190ece74343b3e11b3de00c17cc P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:32.346832 13295 tablet_service.cc:1514] Processing DeleteTablet for tablet b291e3dd6a7a454ca4d736eff8dd3a41 with delete_type TABLET_DATA_TOMBSTONED (TS ba0194098aa84449ae17e8a0d4c43703 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:33250
I20240503 13:56:32.347060 17178 tablet_replica.cc:330] T b291e3dd6a7a454ca4d736eff8dd3a41 P ba0194098aa84449ae17e8a0d4c43703: stopping tablet replica
I20240503 13:56:32.347206 17178 raft_consensus.cc:2229] T b291e3dd6a7a454ca4d736eff8dd3a41 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:32.347358 17178 raft_consensus.cc:2258] T b291e3dd6a7a454ca4d736eff8dd3a41 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:32.347837 17178 ts_tablet_manager.cc:1875] T b291e3dd6a7a454ca4d736eff8dd3a41 P ba0194098aa84449ae17e8a0d4c43703: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:56:32.349970 17178 ts_tablet_manager.cc:1888] T b291e3dd6a7a454ca4d736eff8dd3a41 P ba0194098aa84449ae17e8a0d4c43703: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 13:56:32.350059 17178 log.cc:1196] T b291e3dd6a7a454ca4d736eff8dd3a41 P ba0194098aa84449ae17e8a0d4c43703: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/wal/wals/b291e3dd6a7a454ca4d736eff8dd3a41
I20240503 13:56:32.350534 12899 catalog_manager.cc:4830] TS ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279): tablet b291e3dd6a7a454ca4d736eff8dd3a41 (table rebalance_test_table_9 [id=2fd11e7ec03a4f6891fc18482c1728a6]) successfully deleted
I20240503 13:56:32.429143 15493 raft_consensus.cc:1261] T fd6a3d908840456f82c346c4bf1d8160 P 61b887f0d40d497badeca5604c1f0546 [term 2 FOLLOWER]: Refusing update from remote peer 60ba3ffe3b7746469834cd91c84be14e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 13:56:32.429601 15958 consensus_queue.cc:1035] T fd6a3d908840456f82c346c4bf1d8160 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:32.462711 12910 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalance_test_table_14" } modify_external_catalogs: true
I20240503 13:56:32.462884 12910 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalance_test_table_14" } modify_external_catalogs: true
I20240503 13:56:32.465371 12910 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Sending DeleteTablet for 3 replicas of tablet 327a78731e454b0baa59ad7aa54c322b
I20240503 13:56:32.465699 12910 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Sending DeleteTablet for 3 replicas of tablet d4e32f627d314080b985a65eddc851dd
I20240503 13:56:32.465911 13424 tablet_service.cc:1514] Processing DeleteTablet for tablet 327a78731e454b0baa59ad7aa54c322b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:32 UTC) from {username='slave'} at 127.0.0.1:33678
I20240503 13:56:32.466115 12910 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Sending DeleteTablet for 4 replicas of tablet 866a258ecb2e4685908d5382cb594959
I20240503 13:56:32.466145 13033 tablet_service.cc:1514] Processing DeleteTablet for tablet 327a78731e454b0baa59ad7aa54c322b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:32 UTC) from {username='slave'} at 127.0.0.1:59458
I20240503 13:56:32.466145 13165 tablet_service.cc:1514] Processing DeleteTablet for tablet 327a78731e454b0baa59ad7aa54c322b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:32 UTC) from {username='slave'} at 127.0.0.1:41668
I20240503 13:56:32.466318 16085 tablet_replica.cc:330] T 327a78731e454b0baa59ad7aa54c322b P 60ba3ffe3b7746469834cd91c84be14e: stopping tablet replica
I20240503 13:56:32.466320 16087 tablet_replica.cc:330] T 327a78731e454b0baa59ad7aa54c322b P 7690ee79577c45e3a2c8f8b552c0bff8: stopping tablet replica
I20240503 13:56:32.466459 16087 raft_consensus.cc:2229] T 327a78731e454b0baa59ad7aa54c322b P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:32.466456 16085 raft_consensus.cc:2229] T 327a78731e454b0baa59ad7aa54c322b P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:32.466588 16087 raft_consensus.cc:2258] T 327a78731e454b0baa59ad7aa54c322b P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:32.466593 16085 raft_consensus.cc:2258] T 327a78731e454b0baa59ad7aa54c322b P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:32.466622 13295 tablet_service.cc:1514] Processing DeleteTablet for tablet d4e32f627d314080b985a65eddc851dd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:32 UTC) from {username='slave'} at 127.0.0.1:33250
I20240503 13:56:32.466804 17178 tablet_replica.cc:330] T d4e32f627d314080b985a65eddc851dd P ba0194098aa84449ae17e8a0d4c43703: stopping tablet replica
I20240503 13:56:32.466809 13034 tablet_service.cc:1514] Processing DeleteTablet for tablet 866a258ecb2e4685908d5382cb594959 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:32 UTC) from {username='slave'} at 127.0.0.1:59458
I20240503 13:56:32.466948 17178 raft_consensus.cc:2229] T d4e32f627d314080b985a65eddc851dd P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:56:32.466987 13033 tablet_service.cc:1514] Processing DeleteTablet for tablet d4e32f627d314080b985a65eddc851dd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:32 UTC) from {username='slave'} at 127.0.0.1:59458
I20240503 13:56:32.467105 16087 ts_tablet_manager.cc:1875] T 327a78731e454b0baa59ad7aa54c322b P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:32.467103 16085 ts_tablet_manager.cc:1875] T 327a78731e454b0baa59ad7aa54c322b P 60ba3ffe3b7746469834cd91c84be14e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:32.467166 17178 raft_consensus.cc:2258] T d4e32f627d314080b985a65eddc851dd P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:32.467592 16863 tablet_replica.cc:330] T 327a78731e454b0baa59ad7aa54c322b P 8ccf5a4ac450414c9d3579bd5800b770: stopping tablet replica
I20240503 13:56:32.467671 17178 ts_tablet_manager.cc:1875] T d4e32f627d314080b985a65eddc851dd P ba0194098aa84449ae17e8a0d4c43703: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:32.467720 16863 raft_consensus.cc:2229] T 327a78731e454b0baa59ad7aa54c322b P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:56:32.467895 16863 raft_consensus.cc:2258] T 327a78731e454b0baa59ad7aa54c322b P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:32.468379 16863 ts_tablet_manager.cc:1875] T 327a78731e454b0baa59ad7aa54c322b P 8ccf5a4ac450414c9d3579bd5800b770: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:32.469341 16087 ts_tablet_manager.cc:1888] T 327a78731e454b0baa59ad7aa54c322b P 7690ee79577c45e3a2c8f8b552c0bff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:56:32.469341 16085 ts_tablet_manager.cc:1888] T 327a78731e454b0baa59ad7aa54c322b P 60ba3ffe3b7746469834cd91c84be14e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:56:32.469404 16085 log.cc:1196] T 327a78731e454b0baa59ad7aa54c322b P 60ba3ffe3b7746469834cd91c84be14e: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal/wals/327a78731e454b0baa59ad7aa54c322b
I20240503 13:56:32.469404 16087 log.cc:1196] T 327a78731e454b0baa59ad7aa54c322b P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal/wals/327a78731e454b0baa59ad7aa54c322b
I20240503 13:56:32.469652 16087 ts_tablet_manager.cc:1909] T 327a78731e454b0baa59ad7aa54c322b P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting consensus metadata
I20240503 13:56:32.469652 16085 ts_tablet_manager.cc:1909] T 327a78731e454b0baa59ad7aa54c322b P 60ba3ffe3b7746469834cd91c84be14e: Deleting consensus metadata
I20240503 13:56:32.469846 17178 ts_tablet_manager.cc:1888] T d4e32f627d314080b985a65eddc851dd P ba0194098aa84449ae17e8a0d4c43703: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:56:32.469903 17178 log.cc:1196] T d4e32f627d314080b985a65eddc851dd P ba0194098aa84449ae17e8a0d4c43703: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/wal/wals/d4e32f627d314080b985a65eddc851dd
I20240503 13:56:32.470118 17178 ts_tablet_manager.cc:1909] T d4e32f627d314080b985a65eddc851dd P ba0194098aa84449ae17e8a0d4c43703: Deleting consensus metadata
I20240503 13:56:32.470433 16085 tablet_replica.cc:330] T 866a258ecb2e4685908d5382cb594959 P 60ba3ffe3b7746469834cd91c84be14e: stopping tablet replica
I20240503 13:56:32.470554 16085 raft_consensus.cc:2229] T 866a258ecb2e4685908d5382cb594959 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:32.470649 16085 raft_consensus.cc:2258] T 866a258ecb2e4685908d5382cb594959 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:32.470690 16863 ts_tablet_manager.cc:1888] T 327a78731e454b0baa59ad7aa54c322b P 8ccf5a4ac450414c9d3579bd5800b770: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:56:32.470754 16863 log.cc:1196] T 327a78731e454b0baa59ad7aa54c322b P 8ccf5a4ac450414c9d3579bd5800b770: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/wal/wals/327a78731e454b0baa59ad7aa54c322b
I20240503 13:56:32.470829 12899 catalog_manager.cc:4830] TS ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279): tablet d4e32f627d314080b985a65eddc851dd (table rebalance_test_table_14 [id=31d255eaa071464fb137ad403ae1db26]) successfully deleted
I20240503 13:56:32.470985 16863 ts_tablet_manager.cc:1909] T 327a78731e454b0baa59ad7aa54c322b P 8ccf5a4ac450414c9d3579bd5800b770: Deleting consensus metadata
I20240503 13:56:32.471019 16085 ts_tablet_manager.cc:1875] T 866a258ecb2e4685908d5382cb594959 P 60ba3ffe3b7746469834cd91c84be14e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:32.471458 13165 tablet_service.cc:1514] Processing DeleteTablet for tablet 866a258ecb2e4685908d5382cb594959 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:32 UTC) from {username='slave'} at 127.0.0.1:41668
I20240503 13:56:32.471539 12899 catalog_manager.cc:4830] TS 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387): tablet 327a78731e454b0baa59ad7aa54c322b (table rebalance_test_table_14 [id=31d255eaa071464fb137ad403ae1db26]) successfully deleted
I20240503 13:56:32.471832 16863 tablet_replica.cc:330] T 866a258ecb2e4685908d5382cb594959 P 8ccf5a4ac450414c9d3579bd5800b770: stopping tablet replica
I20240503 13:56:32.472015 16863 raft_consensus.cc:2229] T 866a258ecb2e4685908d5382cb594959 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:32.472201 16863 raft_consensus.cc:2258] T 866a258ecb2e4685908d5382cb594959 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:32.472544 15604 tablet_service.cc:1514] Processing DeleteTablet for tablet 866a258ecb2e4685908d5382cb594959 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:32 UTC) from {username='slave'} at 127.0.0.1:60718
I20240503 13:56:32.472707 16085 ts_tablet_manager.cc:1888] T 866a258ecb2e4685908d5382cb594959 P 60ba3ffe3b7746469834cd91c84be14e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:32.472760 16085 log.cc:1196] T 866a258ecb2e4685908d5382cb594959 P 60ba3ffe3b7746469834cd91c84be14e: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal/wals/866a258ecb2e4685908d5382cb594959
I20240503 13:56:32.472754 13423 tablet_service.cc:1514] Processing DeleteTablet for tablet 866a258ecb2e4685908d5382cb594959 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:32 UTC) from {username='slave'} at 127.0.0.1:33678
I20240503 13:56:32.472860 13424 tablet_service.cc:1514] Processing DeleteTablet for tablet d4e32f627d314080b985a65eddc851dd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:32 UTC) from {username='slave'} at 127.0.0.1:33678
I20240503 13:56:32.472985 16085 ts_tablet_manager.cc:1909] T 866a258ecb2e4685908d5382cb594959 P 60ba3ffe3b7746469834cd91c84be14e: Deleting consensus metadata
I20240503 13:56:32.473001 16863 ts_tablet_manager.cc:1875] T 866a258ecb2e4685908d5382cb594959 P 8ccf5a4ac450414c9d3579bd5800b770: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:32.473260 17340 tablet_replica.cc:330] T 866a258ecb2e4685908d5382cb594959 P 009f849ee7b54c80a3f5264695047e4d: stopping tablet replica
I20240503 13:56:32.473285 16087 tablet_replica.cc:330] T 866a258ecb2e4685908d5382cb594959 P 7690ee79577c45e3a2c8f8b552c0bff8: stopping tablet replica
I20240503 13:56:32.473573 16087 raft_consensus.cc:2229] T 866a258ecb2e4685908d5382cb594959 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:56:32.473735 16085 tablet_replica.cc:330] T d4e32f627d314080b985a65eddc851dd P 60ba3ffe3b7746469834cd91c84be14e: stopping tablet replica
I20240503 13:56:32.473878 16085 raft_consensus.cc:2229] T d4e32f627d314080b985a65eddc851dd P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:32.473927 17340 raft_consensus.cc:2229] T 866a258ecb2e4685908d5382cb594959 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:32.474018 16085 raft_consensus.cc:2258] T d4e32f627d314080b985a65eddc851dd P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:32.474004 16087 raft_consensus.cc:2258] T 866a258ecb2e4685908d5382cb594959 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:32.474117 17340 raft_consensus.cc:2258] T 866a258ecb2e4685908d5382cb594959 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:32.474516 16085 ts_tablet_manager.cc:1875] T d4e32f627d314080b985a65eddc851dd P 60ba3ffe3b7746469834cd91c84be14e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:32.474803 17340 ts_tablet_manager.cc:1875] T 866a258ecb2e4685908d5382cb594959 P 009f849ee7b54c80a3f5264695047e4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:32.475086 16863 ts_tablet_manager.cc:1888] T 866a258ecb2e4685908d5382cb594959 P 8ccf5a4ac450414c9d3579bd5800b770: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:32.475179 16863 log.cc:1196] T 866a258ecb2e4685908d5382cb594959 P 8ccf5a4ac450414c9d3579bd5800b770: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/wal/wals/866a258ecb2e4685908d5382cb594959
I20240503 13:56:32.475440 16863 ts_tablet_manager.cc:1909] T 866a258ecb2e4685908d5382cb594959 P 8ccf5a4ac450414c9d3579bd5800b770: Deleting consensus metadata
I20240503 13:56:32.475823 16087 ts_tablet_manager.cc:1875] T 866a258ecb2e4685908d5382cb594959 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:32.477120 17340 ts_tablet_manager.cc:1888] T 866a258ecb2e4685908d5382cb594959 P 009f849ee7b54c80a3f5264695047e4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:32.477191 17340 log.cc:1196] T 866a258ecb2e4685908d5382cb594959 P 009f849ee7b54c80a3f5264695047e4d: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/wal/wals/866a258ecb2e4685908d5382cb594959
I20240503 13:56:32.477190 16085 ts_tablet_manager.cc:1888] T d4e32f627d314080b985a65eddc851dd P 60ba3ffe3b7746469834cd91c84be14e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:56:32.477291 16085 log.cc:1196] T d4e32f627d314080b985a65eddc851dd P 60ba3ffe3b7746469834cd91c84be14e: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal/wals/d4e32f627d314080b985a65eddc851dd
I20240503 13:56:32.477445 17340 ts_tablet_manager.cc:1909] T 866a258ecb2e4685908d5382cb594959 P 009f849ee7b54c80a3f5264695047e4d: Deleting consensus metadata
I20240503 13:56:32.477569 16085 ts_tablet_manager.cc:1909] T d4e32f627d314080b985a65eddc851dd P 60ba3ffe3b7746469834cd91c84be14e: Deleting consensus metadata
I20240503 13:56:32.472092 12896 catalog_manager.cc:4830] TS 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109): tablet 327a78731e454b0baa59ad7aa54c322b (table rebalance_test_table_14 [id=31d255eaa071464fb137ad403ae1db26]) successfully deleted
I20240503 13:56:32.476044 12899 catalog_manager.cc:4830] TS 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387): tablet 866a258ecb2e4685908d5382cb594959 (table rebalance_test_table_14 [id=31d255eaa071464fb137ad403ae1db26]) successfully deleted
I20240503 13:56:32.478236 12896 catalog_manager.cc:4830] TS 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951): tablet 866a258ecb2e4685908d5382cb594959 (table rebalance_test_table_14 [id=31d255eaa071464fb137ad403ae1db26]) successfully deleted
I20240503 13:56:32.471809 12897 catalog_manager.cc:4830] TS 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695): tablet 327a78731e454b0baa59ad7aa54c322b (table rebalance_test_table_14 [id=31d255eaa071464fb137ad403ae1db26]) successfully deleted
I20240503 13:56:32.478590 16087 ts_tablet_manager.cc:1888] T 866a258ecb2e4685908d5382cb594959 P 7690ee79577c45e3a2c8f8b552c0bff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:32.478631 12897 catalog_manager.cc:4830] TS 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695): tablet 866a258ecb2e4685908d5382cb594959 (table rebalance_test_table_14 [id=31d255eaa071464fb137ad403ae1db26]) successfully deleted
I20240503 13:56:32.478657 16087 log.cc:1196] T 866a258ecb2e4685908d5382cb594959 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal/wals/866a258ecb2e4685908d5382cb594959
I20240503 13:56:32.478734 12897 catalog_manager.cc:4830] TS 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695): tablet d4e32f627d314080b985a65eddc851dd (table rebalance_test_table_14 [id=31d255eaa071464fb137ad403ae1db26]) successfully deleted
I20240503 13:56:32.478883 16087 ts_tablet_manager.cc:1909] T 866a258ecb2e4685908d5382cb594959 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting consensus metadata
I20240503 13:56:32.479566 16087 tablet_replica.cc:330] T d4e32f627d314080b985a65eddc851dd P 7690ee79577c45e3a2c8f8b552c0bff8: stopping tablet replica
I20240503 13:56:32.479697 16087 raft_consensus.cc:2229] T d4e32f627d314080b985a65eddc851dd P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:32.479683 12896 catalog_manager.cc:4830] TS 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109): tablet 866a258ecb2e4685908d5382cb594959 (table rebalance_test_table_14 [id=31d255eaa071464fb137ad403ae1db26]) successfully deleted
I20240503 13:56:32.479826 16087 raft_consensus.cc:2258] T d4e32f627d314080b985a65eddc851dd P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:32.480250 16087 ts_tablet_manager.cc:1875] T d4e32f627d314080b985a65eddc851dd P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:32.481791 16087 ts_tablet_manager.cc:1888] T d4e32f627d314080b985a65eddc851dd P 7690ee79577c45e3a2c8f8b552c0bff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:56:32.481853 16087 log.cc:1196] T d4e32f627d314080b985a65eddc851dd P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal/wals/d4e32f627d314080b985a65eddc851dd
I20240503 13:56:32.482040 16087 ts_tablet_manager.cc:1909] T d4e32f627d314080b985a65eddc851dd P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting consensus metadata
I20240503 13:56:32.482700 12896 catalog_manager.cc:4830] TS 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109): tablet d4e32f627d314080b985a65eddc851dd (table rebalance_test_table_14 [id=31d255eaa071464fb137ad403ae1db26]) successfully deleted
I20240503 13:56:32.501231 13447 raft_consensus.cc:1261] T fd6a3d908840456f82c346c4bf1d8160 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 2 FOLLOWER]: Refusing update from remote peer 60ba3ffe3b7746469834cd91c84be14e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 13:56:32.501636 15958 consensus_queue.cc:1035] T fd6a3d908840456f82c346c4bf1d8160 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:32.504642 13316 raft_consensus.cc:1261] T fd6a3d908840456f82c346c4bf1d8160 P ba0194098aa84449ae17e8a0d4c43703 [term 2 FOLLOWER]: Refusing update from remote peer 60ba3ffe3b7746469834cd91c84be14e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 13:56:32.505039 15332 consensus_queue.cc:1035] T fd6a3d908840456f82c346c4bf1d8160 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:32.554086 15332 consensus_queue.cc:1035] T 05e6080b2da14f8ca6f865820bbe77a4 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:32.557863 17343 consensus_queue.cc:1035] T 05e6080b2da14f8ca6f865820bbe77a4 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:32.563172 16266 raft_consensus.cc:482] T fd2db190ece74343b3e11b3de00c17cc P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:32.563269 16266 raft_consensus.cc:504] T fd2db190ece74343b3e11b3de00c17cc P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:32.563637 16266 leader_election.cc:288] T fd2db190ece74343b3e11b3de00c17cc P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951), 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153)
I20240503 13:56:32.563977 15493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd2db190ece74343b3e11b3de00c17cc" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546" is_pre_election: true
I20240503 13:56:32.563985 15624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd2db190ece74343b3e11b3de00c17cc" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d" is_pre_election: true
I20240503 13:56:32.564116 15493 raft_consensus.cc:2454] T fd2db190ece74343b3e11b3de00c17cc P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 0.
I20240503 13:56:32.564116 15624 raft_consensus.cc:2454] T fd2db190ece74343b3e11b3de00c17cc P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 0.
I20240503 13:56:32.564409 15691 leader_election.cc:302] T fd2db190ece74343b3e11b3de00c17cc P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 76c386bef61a4bd5a907266a5118ffe2; no voters:
I20240503 13:56:32.564673 16266 raft_consensus.cc:2781] T fd2db190ece74343b3e11b3de00c17cc P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:32.564723 16266 raft_consensus.cc:482] T fd2db190ece74343b3e11b3de00c17cc P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:32.564759 16266 raft_consensus.cc:3037] T fd2db190ece74343b3e11b3de00c17cc P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:32.565623 16266 raft_consensus.cc:504] T fd2db190ece74343b3e11b3de00c17cc P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:32.565966 16266 leader_election.cc:288] T fd2db190ece74343b3e11b3de00c17cc P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 election: Requested vote from peers 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951), 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153)
I20240503 13:56:32.566268 15493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd2db190ece74343b3e11b3de00c17cc" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546"
I20240503 13:56:32.566397 15493 raft_consensus.cc:3037] T fd2db190ece74343b3e11b3de00c17cc P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:32.566555 15624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd2db190ece74343b3e11b3de00c17cc" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d"
I20240503 13:56:32.566654 15624 raft_consensus.cc:3037] T fd2db190ece74343b3e11b3de00c17cc P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:32.567348 15493 raft_consensus.cc:2454] T fd2db190ece74343b3e11b3de00c17cc P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 1.
I20240503 13:56:32.567528 15624 raft_consensus.cc:2454] T fd2db190ece74343b3e11b3de00c17cc P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 1.
I20240503 13:56:32.567772 15694 leader_election.cc:302] T fd2db190ece74343b3e11b3de00c17cc P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61b887f0d40d497badeca5604c1f0546, 76c386bef61a4bd5a907266a5118ffe2; no voters:
I20240503 13:56:32.568020 16266 raft_consensus.cc:2781] T fd2db190ece74343b3e11b3de00c17cc P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:32.568156 16266 raft_consensus.cc:686] T fd2db190ece74343b3e11b3de00c17cc P 76c386bef61a4bd5a907266a5118ffe2 [term 1 LEADER]: Becoming Leader. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Running, Role: LEADER
I20240503 13:56:32.568286 16266 consensus_queue.cc:237] T fd2db190ece74343b3e11b3de00c17cc P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:32.570053 12910 catalog_manager.cc:5484] T fd2db190ece74343b3e11b3de00c17cc P 76c386bef61a4bd5a907266a5118ffe2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71). New cstate: current_term: 1 leader_uuid: "76c386bef61a4bd5a907266a5118ffe2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:32.599915 15473 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:56:32.601979 15604 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:32.602188 13033 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:32.604132 15737 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:56:32.605348 13424 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:56:32.616530 13295 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 13:56:32.616602 13165 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:32.644416 15867 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:56:32.676417 12910 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:56:32.680580 15493 raft_consensus.cc:1261] T fd2db190ece74343b3e11b3de00c17cc P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Refusing update from remote peer 76c386bef61a4bd5a907266a5118ffe2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:32.680737 15624 raft_consensus.cc:1261] T fd2db190ece74343b3e11b3de00c17cc P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Refusing update from remote peer 76c386bef61a4bd5a907266a5118ffe2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:32.681165 17350 consensus_queue.cc:1035] T fd2db190ece74343b3e11b3de00c17cc P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:32.681524 16266 consensus_queue.cc:1035] T fd2db190ece74343b3e11b3de00c17cc P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:32.685102 12904 catalog_manager.cc:5576] TS 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951) does not have the latest schema for tablet fd2db190ece74343b3e11b3de00c17cc (table rebalancer_dynamic_table [id=2026ccafb8d246a4ac6bca16a95bb464]). Expected version 1 got 0
I20240503 13:56:32.685099 17381 tablet.cc:1722] T fd2db190ece74343b3e11b3de00c17cc P 76c386bef61a4bd5a907266a5118ffe2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:32.685426 17381 tablet.cc:1620] T fd2db190ece74343b3e11b3de00c17cc P 76c386bef61a4bd5a907266a5118ffe2: MemRowSet was empty: no flush needed.
I20240503 13:56:32.685983 17258 consensus_queue.cc:1035] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:32.686043 12910 catalog_manager.cc:5576] TS 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153) does not have the latest schema for tablet fd2db190ece74343b3e11b3de00c17cc (table rebalancer_dynamic_table [id=2026ccafb8d246a4ac6bca16a95bb464]). Expected version 1 got 0
I20240503 13:56:32.687057 13051 consensus_queue.cc:237] T 4bd42569fd3b4b39b0bd489acebbad10 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:32.688529 15889 raft_consensus.cc:1261] T 4bd42569fd3b4b39b0bd489acebbad10 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Refusing update from remote peer 60ba3ffe3b7746469834cd91c84be14e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:56:32.689052 13185 raft_consensus.cc:1261] T 4bd42569fd3b4b39b0bd489acebbad10 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Refusing update from remote peer 60ba3ffe3b7746469834cd91c84be14e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:56:32.689546 17383 tablet.cc:1722] T fd2db190ece74343b3e11b3de00c17cc P 009f849ee7b54c80a3f5264695047e4d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:32.689859 17383 tablet.cc:1620] T fd2db190ece74343b3e11b3de00c17cc P 009f849ee7b54c80a3f5264695047e4d: MemRowSet was empty: no flush needed.
I20240503 13:56:32.689802 17343 consensus_queue.cc:1035] T 4bd42569fd3b4b39b0bd489acebbad10 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:32.691030 17343 consensus_queue.cc:1035] T 4bd42569fd3b4b39b0bd489acebbad10 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:32.692065 17385 consensus_queue.cc:1035] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:32.694074 12910 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=2026ccafb8d246a4ac6bca16a95bb464] alter complete (version 1)
I20240503 13:56:32.695706 12904 catalog_manager.cc:5576] TS 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951) does not have the latest schema for tablet fd2db190ece74343b3e11b3de00c17cc (table rebalancer_dynamic_table [id=2026ccafb8d246a4ac6bca16a95bb464]). Expected version 1 got 0
I20240503 13:56:32.698436 17343 raft_consensus.cc:2932] T 4bd42569fd3b4b39b0bd489acebbad10 P 60ba3ffe3b7746469834cd91c84be14e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:32.698648 12905 catalog_manager.cc:5576] TS 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153) does not have the latest schema for tablet fd2db190ece74343b3e11b3de00c17cc (table rebalancer_dynamic_table [id=2026ccafb8d246a4ac6bca16a95bb464]). Expected version 1 got 0
I20240503 13:56:32.699612 17391 tablet.cc:1722] T fd2db190ece74343b3e11b3de00c17cc P 61b887f0d40d497badeca5604c1f0546: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:32.699776 15889 raft_consensus.cc:2932] T 4bd42569fd3b4b39b0bd489acebbad10 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:32.700220 17391 tablet.cc:1620] T fd2db190ece74343b3e11b3de00c17cc P 61b887f0d40d497badeca5604c1f0546: MemRowSet was empty: no flush needed.
I20240503 13:56:32.701160 13185 raft_consensus.cc:2932] T 4bd42569fd3b4b39b0bd489acebbad10 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:32.702395 12905 catalog_manager.cc:5484] T 4bd42569fd3b4b39b0bd489acebbad10 P 60ba3ffe3b7746469834cd91c84be14e reported cstate change: config changed from index 4 to 5, VOTER 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68) evicted. New cstate: current_term: 1 leader_uuid: "60ba3ffe3b7746469834cd91c84be14e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:32.706310 13424 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bd42569fd3b4b39b0bd489acebbad10 with delete_type TABLET_DATA_TOMBSTONED (TS 7690ee79577c45e3a2c8f8b552c0bff8 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:33678
I20240503 13:56:32.706573 16087 tablet_replica.cc:330] T 4bd42569fd3b4b39b0bd489acebbad10 P 7690ee79577c45e3a2c8f8b552c0bff8: stopping tablet replica
I20240503 13:56:32.706712 16087 raft_consensus.cc:2229] T 4bd42569fd3b4b39b0bd489acebbad10 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:32.706883 16087 raft_consensus.cc:2258] T 4bd42569fd3b4b39b0bd489acebbad10 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:32.707391 16087 ts_tablet_manager.cc:1875] T 4bd42569fd3b4b39b0bd489acebbad10 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:56:32.709580 16087 ts_tablet_manager.cc:1888] T 4bd42569fd3b4b39b0bd489acebbad10 P 7690ee79577c45e3a2c8f8b552c0bff8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 13:56:32.709643 16087 log.cc:1196] T 4bd42569fd3b4b39b0bd489acebbad10 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal/wals/4bd42569fd3b4b39b0bd489acebbad10
I20240503 13:56:32.710153 12896 catalog_manager.cc:4830] TS 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109): tablet 4bd42569fd3b4b39b0bd489acebbad10 (table rebalance_test_table_18 [id=aa180ed819e24d89be0f5570e50d3c82]) successfully deleted
I20240503 13:56:32.782255 12910 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_extra_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:56:32.786315 15604 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:32.791620 13424 tablet_service.cc:1467] Processing CreateTablet for tablet 855c5fc4cefb48b58533f26d62595465 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=658f6c781f344047b9273985ceb17229]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:32.791940 13424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 855c5fc4cefb48b58533f26d62595465. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:32.791913 13165 tablet_service.cc:1467] Processing CreateTablet for tablet a6a94e83461643d4adfc4abc7dfb74e7 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=658f6c781f344047b9273985ceb17229]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:32.792184 13165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6a94e83461643d4adfc4abc7dfb74e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:32.792999 13164 tablet_service.cc:1467] Processing CreateTablet for tablet eb92a057165f4da7b96d910d847ff761 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=658f6c781f344047b9273985ceb17229]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:32.793238 13164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb92a057165f4da7b96d910d847ff761. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:32.794095 17443 tablet_bootstrap.cc:492] T 855c5fc4cefb48b58533f26d62595465 P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:32.794368 15473 tablet_service.cc:1467] Processing CreateTablet for tablet a6a94e83461643d4adfc4abc7dfb74e7 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=658f6c781f344047b9273985ceb17229]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:32.794648 15473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6a94e83461643d4adfc4abc7dfb74e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:32.794811 13033 tablet_service.cc:1467] Processing CreateTablet for tablet 855c5fc4cefb48b58533f26d62595465 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=658f6c781f344047b9273985ceb17229]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:32.795027 17443 tablet_bootstrap.cc:654] T 855c5fc4cefb48b58533f26d62595465 P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:32.795083 13033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 855c5fc4cefb48b58533f26d62595465. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:32.795313 15867 tablet_service.cc:1467] Processing CreateTablet for tablet a6a94e83461643d4adfc4abc7dfb74e7 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=658f6c781f344047b9273985ceb17229]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:32.795671 15867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6a94e83461643d4adfc4abc7dfb74e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:32.796556 17204 tablet_bootstrap.cc:492] T a6a94e83461643d4adfc4abc7dfb74e7 P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:32.796753 15868 tablet_service.cc:1467] Processing CreateTablet for tablet eb92a057165f4da7b96d910d847ff761 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=658f6c781f344047b9273985ceb17229]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:32.797053 15868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb92a057165f4da7b96d910d847ff761. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:32.797538 17204 tablet_bootstrap.cc:654] T a6a94e83461643d4adfc4abc7dfb74e7 P 61b887f0d40d497badeca5604c1f0546: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:32.798780 17204 tablet_bootstrap.cc:492] T a6a94e83461643d4adfc4abc7dfb74e7 P 61b887f0d40d497badeca5604c1f0546: No bootstrap required, opened a new log
I20240503 13:56:32.798753 12910 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:56:32.798861 17204 ts_tablet_manager.cc:1367] T a6a94e83461643d4adfc4abc7dfb74e7 P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:56:32.799257 17204 raft_consensus.cc:348] T a6a94e83461643d4adfc4abc7dfb74e7 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:32.799357 17204 raft_consensus.cc:374] T a6a94e83461643d4adfc4abc7dfb74e7 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:32.799386 17204 raft_consensus.cc:729] T a6a94e83461643d4adfc4abc7dfb74e7 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:32.799511 15472 tablet_service.cc:1467] Processing CreateTablet for tablet eb92a057165f4da7b96d910d847ff761 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=658f6c781f344047b9273985ceb17229]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:32.799503 17204 consensus_queue.cc:260] T a6a94e83461643d4adfc4abc7dfb74e7 P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:32.800055 17204 ts_tablet_manager.cc:1398] T a6a94e83461643d4adfc4abc7dfb74e7 P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:32.800976 17448 tablet_bootstrap.cc:492] T eb92a057165f4da7b96d910d847ff761 P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:32.799779 15472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb92a057165f4da7b96d910d847ff761. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:32.802631 17448 tablet_bootstrap.cc:654] T eb92a057165f4da7b96d910d847ff761 P e47105a0f46e4511b7112e571688a4c7: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:32.804096 17448 tablet_bootstrap.cc:492] T eb92a057165f4da7b96d910d847ff761 P e47105a0f46e4511b7112e571688a4c7: No bootstrap required, opened a new log
I20240503 13:56:32.804177 17448 ts_tablet_manager.cc:1367] T eb92a057165f4da7b96d910d847ff761 P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:56:32.804824 17448 raft_consensus.cc:348] T eb92a057165f4da7b96d910d847ff761 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:32.804929 17448 raft_consensus.cc:374] T eb92a057165f4da7b96d910d847ff761 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:32.805102 17204 tablet_bootstrap.cc:492] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:32.805563 17444 tablet_bootstrap.cc:492] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:32.806043 17204 tablet_bootstrap.cc:654] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:32.808085 17443 tablet_bootstrap.cc:492] T 855c5fc4cefb48b58533f26d62595465 P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:32.808241 17443 ts_tablet_manager.cc:1367] T 855c5fc4cefb48b58533f26d62595465 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240503 13:56:32.808370 15737 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:56:32.808602 17381 tablet.cc:1722] T fd2db190ece74343b3e11b3de00c17cc P 76c386bef61a4bd5a907266a5118ffe2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:32.808779 17444 tablet_bootstrap.cc:654] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:32.808784 17443 raft_consensus.cc:348] T 855c5fc4cefb48b58533f26d62595465 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:32.808885 17443 raft_consensus.cc:374] T 855c5fc4cefb48b58533f26d62595465 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:32.808915 17443 raft_consensus.cc:729] T 855c5fc4cefb48b58533f26d62595465 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:32.808925 17381 tablet.cc:1620] T fd2db190ece74343b3e11b3de00c17cc P 76c386bef61a4bd5a907266a5118ffe2: MemRowSet was empty: no flush needed.
I20240503 13:56:32.809026 17443 consensus_queue.cc:260] T 855c5fc4cefb48b58533f26d62595465 P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:32.809778 17444 tablet_bootstrap.cc:492] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:32.809859 17444 ts_tablet_manager.cc:1367] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:56:32.810010 17443 ts_tablet_manager.cc:1398] T 855c5fc4cefb48b58533f26d62595465 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:32.810293 17444 raft_consensus.cc:348] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:32.810395 17444 raft_consensus.cc:374] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:32.810427 17444 raft_consensus.cc:729] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:32.810549 17444 consensus_queue.cc:260] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:32.810781 17204 tablet_bootstrap.cc:492] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546: No bootstrap required, opened a new log
I20240503 13:56:32.810808 17444 ts_tablet_manager.cc:1398] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:32.810868 17204 ts_tablet_manager.cc:1367] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 13:56:32.811302 17204 raft_consensus.cc:348] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:32.811422 17204 raft_consensus.cc:374] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:32.811455 17204 raft_consensus.cc:729] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:32.811573 17204 consensus_queue.cc:260] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:32.811847 17204 ts_tablet_manager.cc:1398] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:32.814352 17444 tablet_bootstrap.cc:492] T eb92a057165f4da7b96d910d847ff761 P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:32.814440 17383 tablet.cc:1722] T fd2db190ece74343b3e11b3de00c17cc P 009f849ee7b54c80a3f5264695047e4d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:32.814726 17383 tablet.cc:1620] T fd2db190ece74343b3e11b3de00c17cc P 009f849ee7b54c80a3f5264695047e4d: MemRowSet was empty: no flush needed.
I20240503 13:56:32.814873 16248 consensus_queue.cc:1035] T c8a53b7095e948f988c9151882bfa1bd P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:32.815217 17444 tablet_bootstrap.cc:654] T eb92a057165f4da7b96d910d847ff761 P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:32.817019 16248 consensus_queue.cc:1035] T c8a53b7095e948f988c9151882bfa1bd P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:32.817071 13164 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:32.819213 17448 raft_consensus.cc:729] T eb92a057165f4da7b96d910d847ff761 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:32.819473 17448 consensus_queue.cc:260] T eb92a057165f4da7b96d910d847ff761 P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:32.821120 17448 ts_tablet_manager.cc:1398] T eb92a057165f4da7b96d910d847ff761 P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20240503 13:56:32.831270 16404 raft_consensus.cc:482] T eb92a057165f4da7b96d910d847ff761 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:32.831374 16404 raft_consensus.cc:504] T eb92a057165f4da7b96d910d847ff761 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:32.831410 17391 tablet.cc:1722] T fd2db190ece74343b3e11b3de00c17cc P 61b887f0d40d497badeca5604c1f0546: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:32.831727 17391 tablet.cc:1620] T fd2db190ece74343b3e11b3de00c17cc P 61b887f0d40d497badeca5604c1f0546: MemRowSet was empty: no flush needed.
I20240503 13:56:32.831797 16404 leader_election.cc:288] T eb92a057165f4da7b96d910d847ff761 P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:32.832892 12912 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=2026ccafb8d246a4ac6bca16a95bb464] alter complete (version 2)
I20240503 13:56:32.835371 15493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb92a057165f4da7b96d910d847ff761" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546" is_pre_election: true
I20240503 13:56:32.835512 15493 raft_consensus.cc:2454] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 0.
I20240503 13:56:32.835822 15825 leader_election.cc:302] T eb92a057165f4da7b96d910d847ff761 P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61b887f0d40d497badeca5604c1f0546, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:32.836035 16404 raft_consensus.cc:2781] T eb92a057165f4da7b96d910d847ff761 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:32.836105 16404 raft_consensus.cc:482] T eb92a057165f4da7b96d910d847ff761 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:32.836144 16404 raft_consensus.cc:3037] T eb92a057165f4da7b96d910d847ff761 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:32.837154 16404 raft_consensus.cc:504] T eb92a057165f4da7b96d910d847ff761 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:32.837504 16404 leader_election.cc:288] T eb92a057165f4da7b96d910d847ff761 P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 election: Requested vote from peers 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:32.839888 15493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb92a057165f4da7b96d910d847ff761" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546"
I20240503 13:56:32.840029 15493 raft_consensus.cc:3037] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:32.840363 13185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb92a057165f4da7b96d910d847ff761" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770" is_pre_election: true
I20240503 13:56:32.841053 15493 raft_consensus.cc:2454] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 1.
I20240503 13:56:32.841490 13183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb92a057165f4da7b96d910d847ff761" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770"
I20240503 13:56:32.841537 15825 leader_election.cc:302] T eb92a057165f4da7b96d910d847ff761 P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61b887f0d40d497badeca5604c1f0546, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:32.841701 16404 raft_consensus.cc:2781] T eb92a057165f4da7b96d910d847ff761 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:32.841920 16404 raft_consensus.cc:686] T eb92a057165f4da7b96d910d847ff761 P e47105a0f46e4511b7112e571688a4c7 [term 1 LEADER]: Becoming Leader. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Running, Role: LEADER
W20240503 13:56:32.841887 15825 leader_election.cc:341] T eb92a057165f4da7b96d910d847ff761 P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387): Illegal state: must be running to vote when last-logged opid is not known
I20240503 13:56:32.842077 16404 consensus_queue.cc:237] T eb92a057165f4da7b96d910d847ff761 P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
W20240503 13:56:32.842113 15825 leader_election.cc:341] T eb92a057165f4da7b96d910d847ff761 P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387): Illegal state: must be running to vote when last-logged opid is not known
I20240503 13:56:32.843009 15868 tablet_service.cc:1467] Processing CreateTablet for tablet 855c5fc4cefb48b58533f26d62595465 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=658f6c781f344047b9273985ceb17229]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:32.843544 15868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 855c5fc4cefb48b58533f26d62595465. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:32.843694 13424 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:56:32.844224 13295 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 13:56:32.846837 12909 catalog_manager.cc:5484] T eb92a057165f4da7b96d910d847ff761 P e47105a0f46e4511b7112e571688a4c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e47105a0f46e4511b7112e571688a4c7 (127.6.98.72). New cstate: current_term: 1 leader_uuid: "e47105a0f46e4511b7112e571688a4c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } }
I20240503 13:56:32.849500 15867 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:56:32.864688 17448 tablet_bootstrap.cc:492] T a6a94e83461643d4adfc4abc7dfb74e7 P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:32.865950 17448 tablet_bootstrap.cc:654] T a6a94e83461643d4adfc4abc7dfb74e7 P e47105a0f46e4511b7112e571688a4c7: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:32.867996 17448 tablet_bootstrap.cc:492] T a6a94e83461643d4adfc4abc7dfb74e7 P e47105a0f46e4511b7112e571688a4c7: No bootstrap required, opened a new log
I20240503 13:56:32.868148 17448 ts_tablet_manager.cc:1367] T a6a94e83461643d4adfc4abc7dfb74e7 P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:56:32.868656 17448 raft_consensus.cc:348] T a6a94e83461643d4adfc4abc7dfb74e7 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:32.869205 13033 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:32.871218 17448 raft_consensus.cc:374] T a6a94e83461643d4adfc4abc7dfb74e7 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:32.871316 17448 raft_consensus.cc:729] T a6a94e83461643d4adfc4abc7dfb74e7 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:32.872421 17448 consensus_queue.cc:260] T a6a94e83461643d4adfc4abc7dfb74e7 P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:32.873320 17448 ts_tablet_manager.cc:1398] T a6a94e83461643d4adfc4abc7dfb74e7 P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240503 13:56:32.873517 17448 tablet_bootstrap.cc:492] T 855c5fc4cefb48b58533f26d62595465 P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:32.874974 17448 tablet_bootstrap.cc:654] T 855c5fc4cefb48b58533f26d62595465 P e47105a0f46e4511b7112e571688a4c7: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:32.875777 17446 tablet_bootstrap.cc:492] T 855c5fc4cefb48b58533f26d62595465 P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:32.876550 17448 tablet_bootstrap.cc:492] T 855c5fc4cefb48b58533f26d62595465 P e47105a0f46e4511b7112e571688a4c7: No bootstrap required, opened a new log
I20240503 13:56:32.876633 17448 ts_tablet_manager.cc:1367] T 855c5fc4cefb48b58533f26d62595465 P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:56:32.876652 17446 tablet_bootstrap.cc:654] T 855c5fc4cefb48b58533f26d62595465 P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:32.877070 17448 raft_consensus.cc:348] T 855c5fc4cefb48b58533f26d62595465 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:32.877168 17448 raft_consensus.cc:374] T 855c5fc4cefb48b58533f26d62595465 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:32.877197 17448 raft_consensus.cc:729] T 855c5fc4cefb48b58533f26d62595465 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:32.877305 17448 consensus_queue.cc:260] T 855c5fc4cefb48b58533f26d62595465 P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:32.877638 17448 ts_tablet_manager.cc:1398] T 855c5fc4cefb48b58533f26d62595465 P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:32.881605 15472 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:56:32.883951 17446 tablet_bootstrap.cc:492] T 855c5fc4cefb48b58533f26d62595465 P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:32.884013 17444 tablet_bootstrap.cc:492] T eb92a057165f4da7b96d910d847ff761 P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:32.884150 17444 ts_tablet_manager.cc:1367] T eb92a057165f4da7b96d910d847ff761 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.070s user 0.002s sys 0.000s
I20240503 13:56:32.884151 17446 ts_tablet_manager.cc:1367] T 855c5fc4cefb48b58533f26d62595465 P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20240503 13:56:32.884689 17444 raft_consensus.cc:348] T eb92a057165f4da7b96d910d847ff761 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:32.884804 17444 raft_consensus.cc:374] T eb92a057165f4da7b96d910d847ff761 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:32.884835 17444 raft_consensus.cc:729] T eb92a057165f4da7b96d910d847ff761 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:32.884903 17446 raft_consensus.cc:348] T 855c5fc4cefb48b58533f26d62595465 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:32.884956 17444 consensus_queue.cc:260] T eb92a057165f4da7b96d910d847ff761 P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:32.885056 17446 raft_consensus.cc:374] T 855c5fc4cefb48b58533f26d62595465 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:32.885089 17446 raft_consensus.cc:729] T 855c5fc4cefb48b58533f26d62595465 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:32.885218 17444 ts_tablet_manager.cc:1398] T eb92a057165f4da7b96d910d847ff761 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:32.885197 17446 consensus_queue.cc:260] T 855c5fc4cefb48b58533f26d62595465 P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:32.885433 17446 ts_tablet_manager.cc:1398] T 855c5fc4cefb48b58533f26d62595465 P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:32.891291 16949 raft_consensus.cc:482] T eb92a057165f4da7b96d910d847ff761 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:32.891393 16949 raft_consensus.cc:504] T eb92a057165f4da7b96d910d847ff761 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:32.891777 16949 leader_election.cc:288] T eb92a057165f4da7b96d910d847ff761 P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:32.892172 15493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb92a057165f4da7b96d910d847ff761" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546" is_pre_election: true
I20240503 13:56:32.892354 15493 raft_consensus.cc:2379] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8ccf5a4ac450414c9d3579bd5800b770 in current term 1: Already voted for candidate e47105a0f46e4511b7112e571688a4c7 in this term.
I20240503 13:56:32.892619 15889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb92a057165f4da7b96d910d847ff761" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7" is_pre_election: true
I20240503 13:56:32.895776 13121 leader_election.cc:302] T eb92a057165f4da7b96d910d847ff761 P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ccf5a4ac450414c9d3579bd5800b770; no voters: 61b887f0d40d497badeca5604c1f0546, e47105a0f46e4511b7112e571688a4c7
I20240503 13:56:32.896031 16949 raft_consensus.cc:3037] T eb92a057165f4da7b96d910d847ff761 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:32.897217 16949 raft_consensus.cc:2726] T eb92a057165f4da7b96d910d847ff761 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 13:56:32.903892 17348 raft_consensus.cc:482] T 855c5fc4cefb48b58533f26d62595465 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:32.904006 17348 raft_consensus.cc:504] T 855c5fc4cefb48b58533f26d62595465 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:32.904376 17348 leader_election.cc:288] T 855c5fc4cefb48b58533f26d62595465 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:32.905283 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "855c5fc4cefb48b58533f26d62595465" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
I20240503 13:56:32.905416 13447 raft_consensus.cc:2454] T 855c5fc4cefb48b58533f26d62595465 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 0.
I20240503 13:56:32.905427 15889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "855c5fc4cefb48b58533f26d62595465" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7" is_pre_election: true
I20240503 13:56:32.905556 15889 raft_consensus.cc:2454] T 855c5fc4cefb48b58533f26d62595465 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 0.
I20240503 13:56:32.905928 12987 leader_election.cc:302] T 855c5fc4cefb48b58533f26d62595465 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:32.906105 17348 raft_consensus.cc:2781] T 855c5fc4cefb48b58533f26d62595465 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:32.906165 17348 raft_consensus.cc:482] T 855c5fc4cefb48b58533f26d62595465 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:32.906206 17348 raft_consensus.cc:3037] T 855c5fc4cefb48b58533f26d62595465 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:32.907248 17348 raft_consensus.cc:504] T 855c5fc4cefb48b58533f26d62595465 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:32.907658 17348 leader_election.cc:288] T 855c5fc4cefb48b58533f26d62595465 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 election: Requested vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:32.908025 15889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "855c5fc4cefb48b58533f26d62595465" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7"
I20240503 13:56:32.908141 15889 raft_consensus.cc:3037] T 855c5fc4cefb48b58533f26d62595465 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:32.909124 15889 raft_consensus.cc:2454] T 855c5fc4cefb48b58533f26d62595465 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 1.
I20240503 13:56:32.909409 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "855c5fc4cefb48b58533f26d62595465" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
I20240503 13:56:32.909438 12987 leader_election.cc:302] T 855c5fc4cefb48b58533f26d62595465 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:32.909529 13447 raft_consensus.cc:3037] T 855c5fc4cefb48b58533f26d62595465 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:32.909603 17348 raft_consensus.cc:2781] T 855c5fc4cefb48b58533f26d62595465 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:32.909673 17348 raft_consensus.cc:686] T 855c5fc4cefb48b58533f26d62595465 P 60ba3ffe3b7746469834cd91c84be14e [term 1 LEADER]: Becoming Leader. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Running, Role: LEADER
I20240503 13:56:32.909796 17348 consensus_queue.cc:237] T 855c5fc4cefb48b58533f26d62595465 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:32.910490 13447 raft_consensus.cc:2454] T 855c5fc4cefb48b58533f26d62595465 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 1.
I20240503 13:56:32.918260 12909 catalog_manager.cc:5484] T 855c5fc4cefb48b58533f26d62595465 P 60ba3ffe3b7746469834cd91c84be14e reported cstate change: term changed from 0 to 1, leader changed from <none> to 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65). New cstate: current_term: 1 leader_uuid: "60ba3ffe3b7746469834cd91c84be14e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:32.930409 17457 raft_consensus.cc:482] T a6a94e83461643d4adfc4abc7dfb74e7 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:32.930464 13316 consensus_queue.cc:237] T b47b28a4766a425e8b9bdfea882f979d P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:32.930541 17457 raft_consensus.cc:504] T a6a94e83461643d4adfc4abc7dfb74e7 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:32.930929 17457 leader_election.cc:288] T a6a94e83461643d4adfc4abc7dfb74e7 P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387), 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153)
I20240503 13:56:32.931708 13183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a94e83461643d4adfc4abc7dfb74e7" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770" is_pre_election: true
I20240503 13:56:32.931841 13183 raft_consensus.cc:2454] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 0.
I20240503 13:56:32.932081 15493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a94e83461643d4adfc4abc7dfb74e7" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546" is_pre_election: true
I20240503 13:56:32.932102 15825 leader_election.cc:302] T a6a94e83461643d4adfc4abc7dfb74e7 P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ccf5a4ac450414c9d3579bd5800b770, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:32.932214 15493 raft_consensus.cc:2454] T a6a94e83461643d4adfc4abc7dfb74e7 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 0.
I20240503 13:56:32.932595 13051 raft_consensus.cc:1261] T b47b28a4766a425e8b9bdfea882f979d P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:56:32.932996 15757 raft_consensus.cc:1261] T b47b28a4766a425e8b9bdfea882f979d P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:56:32.933199 17200 consensus_queue.cc:1035] T b47b28a4766a425e8b9bdfea882f979d P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:32.933847 17200 consensus_queue.cc:1035] T b47b28a4766a425e8b9bdfea882f979d P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:32.935052 17457 raft_consensus.cc:2781] T a6a94e83461643d4adfc4abc7dfb74e7 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:32.935114 17457 raft_consensus.cc:482] T a6a94e83461643d4adfc4abc7dfb74e7 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:32.935150 17457 raft_consensus.cc:3037] T a6a94e83461643d4adfc4abc7dfb74e7 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:32.935139 17200 raft_consensus.cc:2932] T b47b28a4766a425e8b9bdfea882f979d P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:32.935497 12909 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:56:32.936172 17457 raft_consensus.cc:504] T a6a94e83461643d4adfc4abc7dfb74e7 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:32.936522 17457 leader_election.cc:288] T a6a94e83461643d4adfc4abc7dfb74e7 P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 election: Requested vote from peers 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387), 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153)
I20240503 13:56:32.937857 12908 catalog_manager.cc:5484] T b47b28a4766a425e8b9bdfea882f979d P ba0194098aa84449ae17e8a0d4c43703 reported cstate change: config changed from index 4 to 5, VOTER 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66) evicted. New cstate: current_term: 1 leader_uuid: "ba0194098aa84449ae17e8a0d4c43703" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:32.939047 15757 raft_consensus.cc:2932] T b47b28a4766a425e8b9bdfea882f979d P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:32.940881 15493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a94e83461643d4adfc4abc7dfb74e7" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546"
I20240503 13:56:32.941009 15493 raft_consensus.cc:3037] T a6a94e83461643d4adfc4abc7dfb74e7 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:32.941978 15493 raft_consensus.cc:2454] T a6a94e83461643d4adfc4abc7dfb74e7 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 1.
I20240503 13:56:32.942332 15825 leader_election.cc:302] T a6a94e83461643d4adfc4abc7dfb74e7 P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61b887f0d40d497badeca5604c1f0546, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:32.942516 17457 raft_consensus.cc:2781] T a6a94e83461643d4adfc4abc7dfb74e7 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:32.942493 17381 tablet.cc:1722] T fd2db190ece74343b3e11b3de00c17cc P 76c386bef61a4bd5a907266a5118ffe2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:32.942595 17457 raft_consensus.cc:686] T a6a94e83461643d4adfc4abc7dfb74e7 P e47105a0f46e4511b7112e571688a4c7 [term 1 LEADER]: Becoming Leader. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Running, Role: LEADER
I20240503 13:56:32.942732 17457 consensus_queue.cc:237] T a6a94e83461643d4adfc4abc7dfb74e7 P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:32.942726 13183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a94e83461643d4adfc4abc7dfb74e7" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770"
I20240503 13:56:32.942849 13183 raft_consensus.cc:3037] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:32.943801 13183 raft_consensus.cc:2454] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 1.
I20240503 13:56:32.946723 17391 tablet.cc:1722] T fd2db190ece74343b3e11b3de00c17cc P 61b887f0d40d497badeca5604c1f0546: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:32.947443 12912 catalog_manager.cc:5484] T a6a94e83461643d4adfc4abc7dfb74e7 P e47105a0f46e4511b7112e571688a4c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e47105a0f46e4511b7112e571688a4c7 (127.6.98.72). New cstate: current_term: 1 leader_uuid: "e47105a0f46e4511b7112e571688a4c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:32.948347 17383 tablet.cc:1722] T fd2db190ece74343b3e11b3de00c17cc P 009f849ee7b54c80a3f5264695047e4d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:32.948549 13164 tablet_service.cc:1514] Processing DeleteTablet for tablet b47b28a4766a425e8b9bdfea882f979d with delete_type TABLET_DATA_TOMBSTONED (TS 8ccf5a4ac450414c9d3579bd5800b770 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:41668
I20240503 13:56:32.949211 16863 tablet_replica.cc:330] T b47b28a4766a425e8b9bdfea882f979d P 8ccf5a4ac450414c9d3579bd5800b770: stopping tablet replica
I20240503 13:56:32.949337 16863 raft_consensus.cc:2229] T b47b28a4766a425e8b9bdfea882f979d P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:32.949465 16863 raft_consensus.cc:2258] T b47b28a4766a425e8b9bdfea882f979d P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:32.949940 16863 ts_tablet_manager.cc:1875] T b47b28a4766a425e8b9bdfea882f979d P 8ccf5a4ac450414c9d3579bd5800b770: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:56:32.951674 13051 raft_consensus.cc:2932] T b47b28a4766a425e8b9bdfea882f979d P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:32.952387 16863 ts_tablet_manager.cc:1888] T b47b28a4766a425e8b9bdfea882f979d P 8ccf5a4ac450414c9d3579bd5800b770: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 13:56:32.952447 16863 log.cc:1196] T b47b28a4766a425e8b9bdfea882f979d P 8ccf5a4ac450414c9d3579bd5800b770: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/wal/wals/b47b28a4766a425e8b9bdfea882f979d
I20240503 13:56:32.952255 12912 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table_new_name [id=2026ccafb8d246a4ac6bca16a95bb464] alter complete (version 3)
I20240503 13:56:32.953589 12899 catalog_manager.cc:4830] TS 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387): tablet b47b28a4766a425e8b9bdfea882f979d (table rebalance_test_table_5 [id=ef32d41e01cb46ed894c4ff29e5c0e52]) successfully deleted
I20240503 13:56:32.972975 12908 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalance_test_table_16" } modify_external_catalogs: true
I20240503 13:56:32.973127 12908 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalance_test_table_16" } modify_external_catalogs: true
I20240503 13:56:32.975778 12908 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Sending DeleteTablet for 3 replicas of tablet 2cb96587807b4437ba9b1b1ca5c95a6a
I20240503 13:56:32.976079 12908 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Sending DeleteTablet for 3 replicas of tablet 9f4d9dca210846edaf7f9bc818c5db29
I20240503 13:56:32.976336 12908 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Sending DeleteTablet for 4 replicas of tablet e9b5dc0762494e6ebdebb7b2833a272e
I20240503 13:56:32.976322 13164 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cb96587807b4437ba9b1b1ca5c95a6a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:32 UTC) from {username='slave'} at 127.0.0.1:41668
I20240503 13:56:32.976596 16863 tablet_replica.cc:330] T 2cb96587807b4437ba9b1b1ca5c95a6a P 8ccf5a4ac450414c9d3579bd5800b770: stopping tablet replica
I20240503 13:56:32.976598 13033 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cb96587807b4437ba9b1b1ca5c95a6a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:32 UTC) from {username='slave'} at 127.0.0.1:59458
I20240503 13:56:32.976758 16085 tablet_replica.cc:330] T 2cb96587807b4437ba9b1b1ca5c95a6a P 60ba3ffe3b7746469834cd91c84be14e: stopping tablet replica
I20240503 13:56:32.976747 16863 raft_consensus.cc:2229] T 2cb96587807b4437ba9b1b1ca5c95a6a P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:32.976821 13295 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cb96587807b4437ba9b1b1ca5c95a6a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:32 UTC) from {username='slave'} at 127.0.0.1:33250
I20240503 13:56:32.976876 16085 raft_consensus.cc:2229] T 2cb96587807b4437ba9b1b1ca5c95a6a P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:32.976904 16863 raft_consensus.cc:2258] T 2cb96587807b4437ba9b1b1ca5c95a6a P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:32.977015 16085 raft_consensus.cc:2258] T 2cb96587807b4437ba9b1b1ca5c95a6a P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:32.977494 16863 ts_tablet_manager.cc:1875] T 2cb96587807b4437ba9b1b1ca5c95a6a P 8ccf5a4ac450414c9d3579bd5800b770: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:32.977485 16085 ts_tablet_manager.cc:1875] T 2cb96587807b4437ba9b1b1ca5c95a6a P 60ba3ffe3b7746469834cd91c84be14e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:32.979755 16863 ts_tablet_manager.cc:1888] T 2cb96587807b4437ba9b1b1ca5c95a6a P 8ccf5a4ac450414c9d3579bd5800b770: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:56:32.979755 16085 ts_tablet_manager.cc:1888] T 2cb96587807b4437ba9b1b1ca5c95a6a P 60ba3ffe3b7746469834cd91c84be14e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:56:32.979840 16863 log.cc:1196] T 2cb96587807b4437ba9b1b1ca5c95a6a P 8ccf5a4ac450414c9d3579bd5800b770: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/wal/wals/2cb96587807b4437ba9b1b1ca5c95a6a
I20240503 13:56:32.979840 16085 log.cc:1196] T 2cb96587807b4437ba9b1b1ca5c95a6a P 60ba3ffe3b7746469834cd91c84be14e: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal/wals/2cb96587807b4437ba9b1b1ca5c95a6a
I20240503 13:56:32.980192 16863 ts_tablet_manager.cc:1909] T 2cb96587807b4437ba9b1b1ca5c95a6a P 8ccf5a4ac450414c9d3579bd5800b770: Deleting consensus metadata
I20240503 13:56:32.980192 16085 ts_tablet_manager.cc:1909] T 2cb96587807b4437ba9b1b1ca5c95a6a P 60ba3ffe3b7746469834cd91c84be14e: Deleting consensus metadata
I20240503 13:56:32.980425 13034 tablet_service.cc:1514] Processing DeleteTablet for tablet e9b5dc0762494e6ebdebb7b2833a272e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:32 UTC) from {username='slave'} at 127.0.0.1:59458
I20240503 13:56:32.980612 13424 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f4d9dca210846edaf7f9bc818c5db29 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:32 UTC) from {username='slave'} at 127.0.0.1:33678
I20240503 13:56:32.980767 13423 tablet_service.cc:1514] Processing DeleteTablet for tablet e9b5dc0762494e6ebdebb7b2833a272e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:32 UTC) from {username='slave'} at 127.0.0.1:33678
I20240503 13:56:32.980795 16087 tablet_replica.cc:330] T 9f4d9dca210846edaf7f9bc818c5db29 P 7690ee79577c45e3a2c8f8b552c0bff8: stopping tablet replica
I20240503 13:56:32.980896 13033 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f4d9dca210846edaf7f9bc818c5db29 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:32 UTC) from {username='slave'} at 127.0.0.1:59458
I20240503 13:56:32.980937 16087 raft_consensus.cc:2229] T 9f4d9dca210846edaf7f9bc818c5db29 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:56:32.981098 15737 tablet_service.cc:1514] Processing DeleteTablet for tablet e9b5dc0762494e6ebdebb7b2833a272e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:32 UTC) from {username='slave'} at 127.0.0.1:44984
I20240503 13:56:32.981140 16087 raft_consensus.cc:2258] T 9f4d9dca210846edaf7f9bc818c5db29 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:32.981287 13164 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f4d9dca210846edaf7f9bc818c5db29 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:32 UTC) from {username='slave'} at 127.0.0.1:41668
I20240503 13:56:32.981549 12899 catalog_manager.cc:4830] TS 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387): tablet 2cb96587807b4437ba9b1b1ca5c95a6a (table rebalance_test_table_16 [id=8817f3bcbcde485f96984e3c87bf3241]) successfully deleted
I20240503 13:56:32.981676 16087 ts_tablet_manager.cc:1875] T 9f4d9dca210846edaf7f9bc818c5db29 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:32.983431 16087 ts_tablet_manager.cc:1888] T 9f4d9dca210846edaf7f9bc818c5db29 P 7690ee79577c45e3a2c8f8b552c0bff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:56:32.983495 16087 log.cc:1196] T 9f4d9dca210846edaf7f9bc818c5db29 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal/wals/9f4d9dca210846edaf7f9bc818c5db29
I20240503 13:56:32.983698 16087 ts_tablet_manager.cc:1909] T 9f4d9dca210846edaf7f9bc818c5db29 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting consensus metadata
I20240503 13:56:32.984047 13295 tablet_service.cc:1514] Processing DeleteTablet for tablet e9b5dc0762494e6ebdebb7b2833a272e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:32 UTC) from {username='slave'} at 127.0.0.1:33250
I20240503 13:56:32.984679 16087 tablet_replica.cc:330] T e9b5dc0762494e6ebdebb7b2833a272e P 7690ee79577c45e3a2c8f8b552c0bff8: stopping tablet replica
I20240503 13:56:32.984764 12896 catalog_manager.cc:4830] TS 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109): tablet 9f4d9dca210846edaf7f9bc818c5db29 (table rebalance_test_table_16 [id=8817f3bcbcde485f96984e3c87bf3241]) successfully deleted
I20240503 13:56:32.984824 16085 tablet_replica.cc:330] T e9b5dc0762494e6ebdebb7b2833a272e P 60ba3ffe3b7746469834cd91c84be14e: stopping tablet replica
I20240503 13:56:32.984958 16085 raft_consensus.cc:2229] T e9b5dc0762494e6ebdebb7b2833a272e P 60ba3ffe3b7746469834cd91c84be14e [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:56:32.984903 16087 raft_consensus.cc:2229] T e9b5dc0762494e6ebdebb7b2833a272e P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:32.985087 12897 catalog_manager.cc:4830] TS 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695): tablet 2cb96587807b4437ba9b1b1ca5c95a6a (table rebalance_test_table_16 [id=8817f3bcbcde485f96984e3c87bf3241]) successfully deleted
I20240503 13:56:32.985232 16085 raft_consensus.cc:2258] T e9b5dc0762494e6ebdebb7b2833a272e P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:32.985287 16087 raft_consensus.cc:2258] T e9b5dc0762494e6ebdebb7b2833a272e P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:32.985682 16085 ts_tablet_manager.cc:1875] T e9b5dc0762494e6ebdebb7b2833a272e P 60ba3ffe3b7746469834cd91c84be14e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:32.985777 16087 ts_tablet_manager.cc:1875] T e9b5dc0762494e6ebdebb7b2833a272e P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:32.986750 16863 tablet_replica.cc:330] T 9f4d9dca210846edaf7f9bc818c5db29 P 8ccf5a4ac450414c9d3579bd5800b770: stopping tablet replica
I20240503 13:56:32.987591 16085 ts_tablet_manager.cc:1888] T e9b5dc0762494e6ebdebb7b2833a272e P 60ba3ffe3b7746469834cd91c84be14e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:32.987681 16085 log.cc:1196] T e9b5dc0762494e6ebdebb7b2833a272e P 60ba3ffe3b7746469834cd91c84be14e: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal/wals/e9b5dc0762494e6ebdebb7b2833a272e
I20240503 13:56:32.987926 16085 ts_tablet_manager.cc:1909] T e9b5dc0762494e6ebdebb7b2833a272e P 60ba3ffe3b7746469834cd91c84be14e: Deleting consensus metadata
I20240503 13:56:32.988004 16087 ts_tablet_manager.cc:1888] T e9b5dc0762494e6ebdebb7b2833a272e P 7690ee79577c45e3a2c8f8b552c0bff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:32.988097 16087 log.cc:1196] T e9b5dc0762494e6ebdebb7b2833a272e P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal/wals/e9b5dc0762494e6ebdebb7b2833a272e
I20240503 13:56:32.988272 16863 raft_consensus.cc:2229] T 9f4d9dca210846edaf7f9bc818c5db29 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:32.988466 16863 raft_consensus.cc:2258] T 9f4d9dca210846edaf7f9bc818c5db29 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:32.988598 16087 ts_tablet_manager.cc:1909] T e9b5dc0762494e6ebdebb7b2833a272e P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting consensus metadata
I20240503 13:56:32.988966 12897 catalog_manager.cc:4830] TS 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695): tablet e9b5dc0762494e6ebdebb7b2833a272e (table rebalance_test_table_16 [id=8817f3bcbcde485f96984e3c87bf3241]) successfully deleted
I20240503 13:56:32.989059 16863 ts_tablet_manager.cc:1875] T 9f4d9dca210846edaf7f9bc818c5db29 P 8ccf5a4ac450414c9d3579bd5800b770: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:32.989234 12896 catalog_manager.cc:4830] TS 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109): tablet e9b5dc0762494e6ebdebb7b2833a272e (table rebalance_test_table_16 [id=8817f3bcbcde485f96984e3c87bf3241]) successfully deleted
I20240503 13:56:32.989456 16085 tablet_replica.cc:330] T 9f4d9dca210846edaf7f9bc818c5db29 P 60ba3ffe3b7746469834cd91c84be14e: stopping tablet replica
I20240503 13:56:32.989583 16085 raft_consensus.cc:2229] T 9f4d9dca210846edaf7f9bc818c5db29 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:32.989708 16085 raft_consensus.cc:2258] T 9f4d9dca210846edaf7f9bc818c5db29 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:32.990139 16085 ts_tablet_manager.cc:1875] T 9f4d9dca210846edaf7f9bc818c5db29 P 60ba3ffe3b7746469834cd91c84be14e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:32.990906 16863 ts_tablet_manager.cc:1888] T 9f4d9dca210846edaf7f9bc818c5db29 P 8ccf5a4ac450414c9d3579bd5800b770: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:56:32.990973 16863 log.cc:1196] T 9f4d9dca210846edaf7f9bc818c5db29 P 8ccf5a4ac450414c9d3579bd5800b770: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/wal/wals/9f4d9dca210846edaf7f9bc818c5db29
I20240503 13:56:32.991181 16863 ts_tablet_manager.cc:1909] T 9f4d9dca210846edaf7f9bc818c5db29 P 8ccf5a4ac450414c9d3579bd5800b770: Deleting consensus metadata
I20240503 13:56:32.991434 17494 tablet_replica.cc:330] T e9b5dc0762494e6ebdebb7b2833a272e P 76c386bef61a4bd5a907266a5118ffe2: stopping tablet replica
I20240503 13:56:32.991621 17494 raft_consensus.cc:2229] T e9b5dc0762494e6ebdebb7b2833a272e P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:32.991780 17494 raft_consensus.cc:2258] T e9b5dc0762494e6ebdebb7b2833a272e P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:32.991926 12899 catalog_manager.cc:4830] TS 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387): tablet 9f4d9dca210846edaf7f9bc818c5db29 (table rebalance_test_table_16 [id=8817f3bcbcde485f96984e3c87bf3241]) successfully deleted
I20240503 13:56:32.992053 16085 ts_tablet_manager.cc:1888] T 9f4d9dca210846edaf7f9bc818c5db29 P 60ba3ffe3b7746469834cd91c84be14e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:56:32.992121 16085 log.cc:1196] T 9f4d9dca210846edaf7f9bc818c5db29 P 60ba3ffe3b7746469834cd91c84be14e: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal/wals/9f4d9dca210846edaf7f9bc818c5db29
I20240503 13:56:32.992342 16085 ts_tablet_manager.cc:1909] T 9f4d9dca210846edaf7f9bc818c5db29 P 60ba3ffe3b7746469834cd91c84be14e: Deleting consensus metadata
I20240503 13:56:32.992378 17494 ts_tablet_manager.cc:1875] T e9b5dc0762494e6ebdebb7b2833a272e P 76c386bef61a4bd5a907266a5118ffe2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:32.993137 12897 catalog_manager.cc:4830] TS 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695): tablet 9f4d9dca210846edaf7f9bc818c5db29 (table rebalance_test_table_16 [id=8817f3bcbcde485f96984e3c87bf3241]) successfully deleted
I20240503 13:56:32.993389 17493 tablet_replica.cc:330] T 2cb96587807b4437ba9b1b1ca5c95a6a P ba0194098aa84449ae17e8a0d4c43703: stopping tablet replica
I20240503 13:56:32.993537 17493 raft_consensus.cc:2229] T 2cb96587807b4437ba9b1b1ca5c95a6a P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:56:32.993726 17493 raft_consensus.cc:2258] T 2cb96587807b4437ba9b1b1ca5c95a6a P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:32.994235 17493 ts_tablet_manager.cc:1875] T 2cb96587807b4437ba9b1b1ca5c95a6a P ba0194098aa84449ae17e8a0d4c43703: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:32.996176 17493 ts_tablet_manager.cc:1888] T 2cb96587807b4437ba9b1b1ca5c95a6a P ba0194098aa84449ae17e8a0d4c43703: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:56:32.996243 17493 log.cc:1196] T 2cb96587807b4437ba9b1b1ca5c95a6a P ba0194098aa84449ae17e8a0d4c43703: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/wal/wals/2cb96587807b4437ba9b1b1ca5c95a6a
I20240503 13:56:32.996474 17493 ts_tablet_manager.cc:1909] T 2cb96587807b4437ba9b1b1ca5c95a6a P ba0194098aa84449ae17e8a0d4c43703: Deleting consensus metadata
I20240503 13:56:32.997931 17494 ts_tablet_manager.cc:1888] T e9b5dc0762494e6ebdebb7b2833a272e P 76c386bef61a4bd5a907266a5118ffe2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:32.998050 17494 log.cc:1196] T e9b5dc0762494e6ebdebb7b2833a272e P 76c386bef61a4bd5a907266a5118ffe2: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/wal/wals/e9b5dc0762494e6ebdebb7b2833a272e
I20240503 13:56:32.998302 17494 ts_tablet_manager.cc:1909] T e9b5dc0762494e6ebdebb7b2833a272e P 76c386bef61a4bd5a907266a5118ffe2: Deleting consensus metadata
I20240503 13:56:32.999495 17493 tablet_replica.cc:330] T e9b5dc0762494e6ebdebb7b2833a272e P ba0194098aa84449ae17e8a0d4c43703: stopping tablet replica
I20240503 13:56:32.999678 17493 raft_consensus.cc:2229] T e9b5dc0762494e6ebdebb7b2833a272e P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:32.999745 12899 catalog_manager.cc:4830] TS ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279): tablet 2cb96587807b4437ba9b1b1ca5c95a6a (table rebalance_test_table_16 [id=8817f3bcbcde485f96984e3c87bf3241]) successfully deleted
I20240503 13:56:32.999928 17493 raft_consensus.cc:2258] T e9b5dc0762494e6ebdebb7b2833a272e P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:33.000566 17493 ts_tablet_manager.cc:1875] T e9b5dc0762494e6ebdebb7b2833a272e P ba0194098aa84449ae17e8a0d4c43703: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:33.000862 12899 catalog_manager.cc:4830] TS 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885): tablet e9b5dc0762494e6ebdebb7b2833a272e (table rebalance_test_table_16 [id=8817f3bcbcde485f96984e3c87bf3241]) successfully deleted
I20240503 13:56:33.002601 17493 ts_tablet_manager.cc:1888] T e9b5dc0762494e6ebdebb7b2833a272e P ba0194098aa84449ae17e8a0d4c43703: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:33.002665 17493 log.cc:1196] T e9b5dc0762494e6ebdebb7b2833a272e P ba0194098aa84449ae17e8a0d4c43703: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/wal/wals/e9b5dc0762494e6ebdebb7b2833a272e
I20240503 13:56:33.002892 17493 ts_tablet_manager.cc:1909] T e9b5dc0762494e6ebdebb7b2833a272e P ba0194098aa84449ae17e8a0d4c43703: Deleting consensus metadata
I20240503 13:56:33.003592 12899 catalog_manager.cc:4830] TS ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279): tablet e9b5dc0762494e6ebdebb7b2833a272e (table rebalance_test_table_16 [id=8817f3bcbcde485f96984e3c87bf3241]) successfully deleted
I20240503 13:56:33.050271 15472 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:56:33.050559 13423 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 13:56:33.054440 12903 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:33.054627 12903 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:33.056129 13033 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:33.062211 12903 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Sending DeleteTablet for 3 replicas of tablet fd2db190ece74343b3e11b3de00c17cc
I20240503 13:56:33.062873 15737 tablet_service.cc:1514] Processing DeleteTablet for tablet fd2db190ece74343b3e11b3de00c17cc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:33 UTC) from {username='slave'} at 127.0.0.1:44984
I20240503 13:56:33.063314 15604 tablet_service.cc:1514] Processing DeleteTablet for tablet fd2db190ece74343b3e11b3de00c17cc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:33 UTC) from {username='slave'} at 127.0.0.1:60718
I20240503 13:56:33.063565 17494 tablet_replica.cc:330] T fd2db190ece74343b3e11b3de00c17cc P 76c386bef61a4bd5a907266a5118ffe2: stopping tablet replica
I20240503 13:56:33.063706 17494 raft_consensus.cc:2229] T fd2db190ece74343b3e11b3de00c17cc P 76c386bef61a4bd5a907266a5118ffe2 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:56:33.063894 17494 raft_consensus.cc:2258] T fd2db190ece74343b3e11b3de00c17cc P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:33.063941 15472 tablet_service.cc:1514] Processing DeleteTablet for tablet fd2db190ece74343b3e11b3de00c17cc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:33 UTC) from {username='slave'} at 127.0.0.1:58758
I20240503 13:56:33.064862 17494 ts_tablet_manager.cc:1875] T fd2db190ece74343b3e11b3de00c17cc P 76c386bef61a4bd5a907266a5118ffe2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:33.065074 17535 tablet_replica.cc:330] T fd2db190ece74343b3e11b3de00c17cc P 61b887f0d40d497badeca5604c1f0546: stopping tablet replica
I20240503 13:56:33.065210 17535 raft_consensus.cc:2229] T fd2db190ece74343b3e11b3de00c17cc P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:33.065279 15604 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:33.065326 17535 raft_consensus.cc:2258] T fd2db190ece74343b3e11b3de00c17cc P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:33.066206 17535 ts_tablet_manager.cc:1875] T fd2db190ece74343b3e11b3de00c17cc P 61b887f0d40d497badeca5604c1f0546: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:33.067090 17494 ts_tablet_manager.cc:1888] T fd2db190ece74343b3e11b3de00c17cc P 76c386bef61a4bd5a907266a5118ffe2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:33.067152 17494 log.cc:1196] T fd2db190ece74343b3e11b3de00c17cc P 76c386bef61a4bd5a907266a5118ffe2: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/wal/wals/fd2db190ece74343b3e11b3de00c17cc
I20240503 13:56:33.067392 17494 ts_tablet_manager.cc:1909] T fd2db190ece74343b3e11b3de00c17cc P 76c386bef61a4bd5a907266a5118ffe2: Deleting consensus metadata
I20240503 13:56:33.067896 15737 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:56:33.069391 17535 ts_tablet_manager.cc:1888] T fd2db190ece74343b3e11b3de00c17cc P 61b887f0d40d497badeca5604c1f0546: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:33.069512 17535 log.cc:1196] T fd2db190ece74343b3e11b3de00c17cc P 61b887f0d40d497badeca5604c1f0546: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/wal/wals/fd2db190ece74343b3e11b3de00c17cc
I20240503 13:56:33.070685 17535 ts_tablet_manager.cc:1909] T fd2db190ece74343b3e11b3de00c17cc P 61b887f0d40d497badeca5604c1f0546: Deleting consensus metadata
I20240503 13:56:33.073936 12899 catalog_manager.cc:4830] TS 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153): tablet fd2db190ece74343b3e11b3de00c17cc (table rebalancer_dynamic_table_new_name [id=2026ccafb8d246a4ac6bca16a95bb464]) successfully deleted
I20240503 13:56:33.074785 12899 catalog_manager.cc:4830] TS 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885): tablet fd2db190ece74343b3e11b3de00c17cc (table rebalancer_dynamic_table_new_name [id=2026ccafb8d246a4ac6bca16a95bb464]) successfully deleted
I20240503 13:56:33.075331 13164 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:33.077319 15867 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:56:33.080083 13295 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 13:56:33.114379 17534 tablet_replica.cc:330] T fd2db190ece74343b3e11b3de00c17cc P 009f849ee7b54c80a3f5264695047e4d: stopping tablet replica
I20240503 13:56:33.114683 17534 raft_consensus.cc:2229] T fd2db190ece74343b3e11b3de00c17cc P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:33.114914 17534 raft_consensus.cc:2258] T fd2db190ece74343b3e11b3de00c17cc P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:33.115700 17534 ts_tablet_manager.cc:1875] T fd2db190ece74343b3e11b3de00c17cc P 009f849ee7b54c80a3f5264695047e4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:33.117941 17534 ts_tablet_manager.cc:1888] T fd2db190ece74343b3e11b3de00c17cc P 009f849ee7b54c80a3f5264695047e4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:33.118083 17534 log.cc:1196] T fd2db190ece74343b3e11b3de00c17cc P 009f849ee7b54c80a3f5264695047e4d: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/wal/wals/fd2db190ece74343b3e11b3de00c17cc
I20240503 13:56:33.118377 17534 ts_tablet_manager.cc:1909] T fd2db190ece74343b3e11b3de00c17cc P 009f849ee7b54c80a3f5264695047e4d: Deleting consensus metadata
I20240503 13:56:33.119285 12896 catalog_manager.cc:4830] TS 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951): tablet fd2db190ece74343b3e11b3de00c17cc (table rebalancer_dynamic_table_new_name [id=2026ccafb8d246a4ac6bca16a95bb464]) successfully deleted
I20240503 13:56:33.141546 13316 consensus_queue.cc:237] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:33.142810 13183 raft_consensus.cc:1261] T bb4f0bb47a29437090750c4486df2f02 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:56:33.142830 15889 raft_consensus.cc:1261] T bb4f0bb47a29437090750c4486df2f02 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:56:33.143194 16890 consensus_queue.cc:1035] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:33.143370 17200 consensus_queue.cc:1035] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:33.145115 17555 raft_consensus.cc:2932] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:33.145659 13183 raft_consensus.cc:2932] T bb4f0bb47a29437090750c4486df2f02 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:33.145670 15889 raft_consensus.cc:2932] T bb4f0bb47a29437090750c4486df2f02 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:33.148357 12911 catalog_manager.cc:5484] T bb4f0bb47a29437090750c4486df2f02 P e47105a0f46e4511b7112e571688a4c7 reported cstate change: config changed from index 4 to 5, VOTER 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68) evicted. New cstate: current_term: 1 leader_uuid: "ba0194098aa84449ae17e8a0d4c43703" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:33.151623 13423 tablet_service.cc:1514] Processing DeleteTablet for tablet bb4f0bb47a29437090750c4486df2f02 with delete_type TABLET_DATA_TOMBSTONED (TS 7690ee79577c45e3a2c8f8b552c0bff8 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:33678
I20240503 13:56:33.151872 16087 tablet_replica.cc:330] T bb4f0bb47a29437090750c4486df2f02 P 7690ee79577c45e3a2c8f8b552c0bff8: stopping tablet replica
I20240503 13:56:33.152011 16087 raft_consensus.cc:2229] T bb4f0bb47a29437090750c4486df2f02 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:33.152156 16087 raft_consensus.cc:2258] T bb4f0bb47a29437090750c4486df2f02 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:33.152724 16087 ts_tablet_manager.cc:1875] T bb4f0bb47a29437090750c4486df2f02 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:56:33.154700 16087 ts_tablet_manager.cc:1888] T bb4f0bb47a29437090750c4486df2f02 P 7690ee79577c45e3a2c8f8b552c0bff8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 13:56:33.154750 16087 log.cc:1196] T bb4f0bb47a29437090750c4486df2f02 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal/wals/bb4f0bb47a29437090750c4486df2f02
I20240503 13:56:33.155196 12896 catalog_manager.cc:4830] TS 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109): tablet bb4f0bb47a29437090750c4486df2f02 (table rebalance_test_table_5 [id=ef32d41e01cb46ed894c4ff29e5c0e52]) successfully deleted
I20240503 13:56:33.164173 12903 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:56:33.169138 13164 tablet_service.cc:1467] Processing CreateTablet for tablet 9f973b16bcb04791ba29d5e3f4fe37a6 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b043b5f53d944ef7822ca72165b0099e]), partition=
I20240503 13:56:33.169160 15737 tablet_service.cc:1467] Processing CreateTablet for tablet 9f973b16bcb04791ba29d5e3f4fe37a6 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b043b5f53d944ef7822ca72165b0099e]), partition=
I20240503 13:56:33.169391 13164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f973b16bcb04791ba29d5e3f4fe37a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:33.169399 15737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f973b16bcb04791ba29d5e3f4fe37a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:33.171489 17444 tablet_bootstrap.cc:492] T 9f973b16bcb04791ba29d5e3f4fe37a6 P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:33.172303 17444 tablet_bootstrap.cc:654] T 9f973b16bcb04791ba29d5e3f4fe37a6 P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:33.173398 17444 tablet_bootstrap.cc:492] T 9f973b16bcb04791ba29d5e3f4fe37a6 P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:33.173473 17444 ts_tablet_manager.cc:1367] T 9f973b16bcb04791ba29d5e3f4fe37a6 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:56:33.173617 17567 tablet_bootstrap.cc:492] T 9f973b16bcb04791ba29d5e3f4fe37a6 P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:33.173736 15867 tablet_service.cc:1467] Processing CreateTablet for tablet 9f973b16bcb04791ba29d5e3f4fe37a6 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b043b5f53d944ef7822ca72165b0099e]), partition=
I20240503 13:56:33.173871 17444 raft_consensus.cc:348] T 9f973b16bcb04791ba29d5e3f4fe37a6 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:33.173959 15867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f973b16bcb04791ba29d5e3f4fe37a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:33.173964 17444 raft_consensus.cc:374] T 9f973b16bcb04791ba29d5e3f4fe37a6 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:33.173991 17444 raft_consensus.cc:729] T 9f973b16bcb04791ba29d5e3f4fe37a6 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:33.174091 17444 consensus_queue.cc:260] T 9f973b16bcb04791ba29d5e3f4fe37a6 P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:33.174350 17444 ts_tablet_manager.cc:1398] T 9f973b16bcb04791ba29d5e3f4fe37a6 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:33.176175 17448 tablet_bootstrap.cc:492] T 9f973b16bcb04791ba29d5e3f4fe37a6 P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:33.177096 17448 tablet_bootstrap.cc:654] T 9f973b16bcb04791ba29d5e3f4fe37a6 P e47105a0f46e4511b7112e571688a4c7: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:33.178064 17567 tablet_bootstrap.cc:654] T 9f973b16bcb04791ba29d5e3f4fe37a6 P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:33.178939 17448 tablet_bootstrap.cc:492] T 9f973b16bcb04791ba29d5e3f4fe37a6 P e47105a0f46e4511b7112e571688a4c7: No bootstrap required, opened a new log
I20240503 13:56:33.179018 17448 ts_tablet_manager.cc:1367] T 9f973b16bcb04791ba29d5e3f4fe37a6 P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 13:56:33.179406 17448 raft_consensus.cc:348] T 9f973b16bcb04791ba29d5e3f4fe37a6 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:33.179495 17448 raft_consensus.cc:374] T 9f973b16bcb04791ba29d5e3f4fe37a6 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:33.179546 17448 raft_consensus.cc:729] T 9f973b16bcb04791ba29d5e3f4fe37a6 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:33.179659 17448 consensus_queue.cc:260] T 9f973b16bcb04791ba29d5e3f4fe37a6 P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:33.179893 17448 ts_tablet_manager.cc:1398] T 9f973b16bcb04791ba29d5e3f4fe37a6 P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:33.181152 17567 tablet_bootstrap.cc:492] T 9f973b16bcb04791ba29d5e3f4fe37a6 P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:33.181283 17567 ts_tablet_manager.cc:1367] T 9f973b16bcb04791ba29d5e3f4fe37a6 P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 13:56:33.181841 17567 raft_consensus.cc:348] T 9f973b16bcb04791ba29d5e3f4fe37a6 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:33.181957 17567 raft_consensus.cc:374] T 9f973b16bcb04791ba29d5e3f4fe37a6 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:33.182024 17567 raft_consensus.cc:729] T 9f973b16bcb04791ba29d5e3f4fe37a6 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:33.182219 17567 consensus_queue.cc:260] T 9f973b16bcb04791ba29d5e3f4fe37a6 P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:33.182515 17567 ts_tablet_manager.cc:1398] T 9f973b16bcb04791ba29d5e3f4fe37a6 P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:33.321542 16266 raft_consensus.cc:482] T 9f973b16bcb04791ba29d5e3f4fe37a6 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:33.321666 16266 raft_consensus.cc:504] T 9f973b16bcb04791ba29d5e3f4fe37a6 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:33.322150 16266 leader_election.cc:288] T 9f973b16bcb04791ba29d5e3f4fe37a6 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:33.323009 15889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f973b16bcb04791ba29d5e3f4fe37a6" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7" is_pre_election: true
I20240503 13:56:33.323180 15889 raft_consensus.cc:2454] T 9f973b16bcb04791ba29d5e3f4fe37a6 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 0.
I20240503 13:56:33.323562 15691 leader_election.cc:302] T 9f973b16bcb04791ba29d5e3f4fe37a6 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76c386bef61a4bd5a907266a5118ffe2, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:33.323760 16266 raft_consensus.cc:2781] T 9f973b16bcb04791ba29d5e3f4fe37a6 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:33.323825 16266 raft_consensus.cc:482] T 9f973b16bcb04791ba29d5e3f4fe37a6 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:33.323860 16266 raft_consensus.cc:3037] T 9f973b16bcb04791ba29d5e3f4fe37a6 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:33.324807 16266 raft_consensus.cc:504] T 9f973b16bcb04791ba29d5e3f4fe37a6 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:33.325143 16266 leader_election.cc:288] T 9f973b16bcb04791ba29d5e3f4fe37a6 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 election: Requested vote from peers 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:33.325518 15889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f973b16bcb04791ba29d5e3f4fe37a6" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7"
I20240503 13:56:33.325645 15889 raft_consensus.cc:3037] T 9f973b16bcb04791ba29d5e3f4fe37a6 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:33.326434 15889 raft_consensus.cc:2454] T 9f973b16bcb04791ba29d5e3f4fe37a6 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 1.
I20240503 13:56:33.326594 13183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f973b16bcb04791ba29d5e3f4fe37a6" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770" is_pre_election: true
I20240503 13:56:33.326632 13185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f973b16bcb04791ba29d5e3f4fe37a6" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770"
I20240503 13:56:33.326730 13183 raft_consensus.cc:2454] T 9f973b16bcb04791ba29d5e3f4fe37a6 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 0.
I20240503 13:56:33.326753 15691 leader_election.cc:302] T 9f973b16bcb04791ba29d5e3f4fe37a6 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76c386bef61a4bd5a907266a5118ffe2, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:33.327108 16266 raft_consensus.cc:2781] T 9f973b16bcb04791ba29d5e3f4fe37a6 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:33.327188 16266 raft_consensus.cc:686] T 9f973b16bcb04791ba29d5e3f4fe37a6 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 LEADER]: Becoming Leader. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Running, Role: LEADER
I20240503 13:56:33.327318 16266 consensus_queue.cc:237] T 9f973b16bcb04791ba29d5e3f4fe37a6 P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:33.329053 12903 catalog_manager.cc:5484] T 9f973b16bcb04791ba29d5e3f4fe37a6 P 76c386bef61a4bd5a907266a5118ffe2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71). New cstate: current_term: 1 leader_uuid: "76c386bef61a4bd5a907266a5118ffe2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:33.344084 13033 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:33.345656 15604 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:56:33.350106 13164 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:33.355583 15737 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:56:33.369042 13423 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 13:56:33.371505 13295 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 13:56:33.382493 17343 consensus_queue.cc:1035] T 855c5fc4cefb48b58533f26d62595465 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:33.386567 15867 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:56:33.387480 15472 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:56:33.394095 17457 consensus_queue.cc:1035] T eb92a057165f4da7b96d910d847ff761 P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:33.394335 17457 consensus_queue.cc:1035] T a6a94e83461643d4adfc4abc7dfb74e7 P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:33.411103 17348 consensus_queue.cc:1035] T 855c5fc4cefb48b58533f26d62595465 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:33.415598 17457 consensus_queue.cc:1035] T eb92a057165f4da7b96d910d847ff761 P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:33.419147 17457 consensus_queue.cc:1035] T a6a94e83461643d4adfc4abc7dfb74e7 P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:33.441886 12907 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:56:33.446122 13316 consensus_queue.cc:237] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:33.446256 15888 raft_consensus.cc:1261] T 9f973b16bcb04791ba29d5e3f4fe37a6 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Refusing update from remote peer 76c386bef61a4bd5a907266a5118ffe2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:33.446512 13185 raft_consensus.cc:3037] T 9f973b16bcb04791ba29d5e3f4fe37a6 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:33.447435 13183 raft_consensus.cc:1261] T a746bcb2d718433f8c9acc8bd027a429 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:56:33.447758 13185 raft_consensus.cc:1261] T 9f973b16bcb04791ba29d5e3f4fe37a6 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Refusing update from remote peer 76c386bef61a4bd5a907266a5118ffe2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:33.448019 15888 raft_consensus.cc:1261] T a746bcb2d718433f8c9acc8bd027a429 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:56:33.448009 17555 consensus_queue.cc:1035] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:33.448338 17555 consensus_queue.cc:1035] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:33.448397 16266 consensus_queue.cc:1035] T 9f973b16bcb04791ba29d5e3f4fe37a6 P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:33.449944 16266 consensus_queue.cc:1035] T 9f973b16bcb04791ba29d5e3f4fe37a6 P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:33.450618 12907 catalog_manager.cc:5576] TS e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753) does not have the latest schema for tablet 9f973b16bcb04791ba29d5e3f4fe37a6 (table rebalancer_dynamic_table [id=b043b5f53d944ef7822ca72165b0099e]). Expected version 1 got 0
I20240503 13:56:33.452046 17645 tablet.cc:1722] T 9f973b16bcb04791ba29d5e3f4fe37a6 P 76c386bef61a4bd5a907266a5118ffe2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:33.452378 17645 tablet.cc:1620] T 9f973b16bcb04791ba29d5e3f4fe37a6 P 76c386bef61a4bd5a907266a5118ffe2: MemRowSet was empty: no flush needed.
I20240503 13:56:33.452404 17555 raft_consensus.cc:2932] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:33.452570 12903 catalog_manager.cc:5576] TS 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387) does not have the latest schema for tablet 9f973b16bcb04791ba29d5e3f4fe37a6 (table rebalancer_dynamic_table [id=b043b5f53d944ef7822ca72165b0099e]). Expected version 1 got 0
I20240503 13:56:33.453902 12903 catalog_manager.cc:5576] TS e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753) does not have the latest schema for tablet 9f973b16bcb04791ba29d5e3f4fe37a6 (table rebalancer_dynamic_table [id=b043b5f53d944ef7822ca72165b0099e]). Expected version 1 got 0
I20240503 13:56:33.454478 17646 tablet.cc:1722] T 9f973b16bcb04791ba29d5e3f4fe37a6 P e47105a0f46e4511b7112e571688a4c7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:33.454795 17646 tablet.cc:1620] T 9f973b16bcb04791ba29d5e3f4fe37a6 P e47105a0f46e4511b7112e571688a4c7: MemRowSet was empty: no flush needed.
I20240503 13:56:33.456759 12903 catalog_manager.cc:5484] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 reported cstate change: config changed from index 4 to 5, VOTER 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68) evicted. New cstate: current_term: 1 leader_uuid: "ba0194098aa84449ae17e8a0d4c43703" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:33.457562 13185 raft_consensus.cc:2932] T a746bcb2d718433f8c9acc8bd027a429 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:33.457669 12905 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_extra_table_12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:56:33.458424 15889 raft_consensus.cc:2932] T a746bcb2d718433f8c9acc8bd027a429 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:33.463074 12908 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=b043b5f53d944ef7822ca72165b0099e] alter complete (version 1)
I20240503 13:56:33.463032 17648 tablet.cc:1722] T 9f973b16bcb04791ba29d5e3f4fe37a6 P 8ccf5a4ac450414c9d3579bd5800b770: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:33.464299 17648 tablet.cc:1620] T 9f973b16bcb04791ba29d5e3f4fe37a6 P 8ccf5a4ac450414c9d3579bd5800b770: MemRowSet was empty: no flush needed.
I20240503 13:56:33.464809 13423 tablet_service.cc:1514] Processing DeleteTablet for tablet a746bcb2d718433f8c9acc8bd027a429 with delete_type TABLET_DATA_TOMBSTONED (TS 7690ee79577c45e3a2c8f8b552c0bff8 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:33678
I20240503 13:56:33.465078 16087 tablet_replica.cc:330] T a746bcb2d718433f8c9acc8bd027a429 P 7690ee79577c45e3a2c8f8b552c0bff8: stopping tablet replica
I20240503 13:56:33.465225 16087 raft_consensus.cc:2229] T a746bcb2d718433f8c9acc8bd027a429 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:33.465368 16087 raft_consensus.cc:2258] T a746bcb2d718433f8c9acc8bd027a429 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:33.465896 16087 ts_tablet_manager.cc:1875] T a746bcb2d718433f8c9acc8bd027a429 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:56:33.468048 16087 ts_tablet_manager.cc:1888] T a746bcb2d718433f8c9acc8bd027a429 P 7690ee79577c45e3a2c8f8b552c0bff8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 13:56:33.468109 16087 log.cc:1196] T a746bcb2d718433f8c9acc8bd027a429 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal/wals/a746bcb2d718433f8c9acc8bd027a429
I20240503 13:56:33.468094 13033 tablet_service.cc:1467] Processing CreateTablet for tablet 3f5d536cffe742e69f4b0d3b93caf76c (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=46a3f42046eb4cb89a40f991a54af62b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:33.468358 13033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f5d536cffe742e69f4b0d3b93caf76c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:33.469451 15737 tablet_service.cc:1467] Processing CreateTablet for tablet 3f5d536cffe742e69f4b0d3b93caf76c (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=46a3f42046eb4cb89a40f991a54af62b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:33.469554 13164 tablet_service.cc:1467] Processing CreateTablet for tablet cbd6d38ea18b484d80f4a918148efd3a (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=46a3f42046eb4cb89a40f991a54af62b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:33.469744 15737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f5d536cffe742e69f4b0d3b93caf76c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:33.469780 13164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbd6d38ea18b484d80f4a918148efd3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:33.470675 17654 tablet_bootstrap.cc:492] T 3f5d536cffe742e69f4b0d3b93caf76c P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:33.470746 15738 tablet_service.cc:1467] Processing CreateTablet for tablet cbd6d38ea18b484d80f4a918148efd3a (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=46a3f42046eb4cb89a40f991a54af62b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:33.470966 15738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbd6d38ea18b484d80f4a918148efd3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:33.471597 17654 tablet_bootstrap.cc:654] T 3f5d536cffe742e69f4b0d3b93caf76c P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:33.471765 17444 tablet_bootstrap.cc:492] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:33.471937 15472 tablet_service.cc:1467] Processing CreateTablet for tablet cbd6d38ea18b484d80f4a918148efd3a (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=46a3f42046eb4cb89a40f991a54af62b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:33.472173 15472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbd6d38ea18b484d80f4a918148efd3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:33.472661 17444 tablet_bootstrap.cc:654] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:33.472658 17567 tablet_bootstrap.cc:492] T 3f5d536cffe742e69f4b0d3b93caf76c P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:33.473508 17567 tablet_bootstrap.cc:654] T 3f5d536cffe742e69f4b0d3b93caf76c P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:33.473912 17444 tablet_bootstrap.cc:492] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:33.473994 17444 ts_tablet_manager.cc:1367] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:56:33.474391 15736 tablet_service.cc:1467] Processing CreateTablet for tablet f48d92757c8a4b8d99910c5863d8fb89 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=46a3f42046eb4cb89a40f991a54af62b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:33.474447 17444 raft_consensus.cc:348] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:33.474560 17444 raft_consensus.cc:374] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:33.474587 15736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f48d92757c8a4b8d99910c5863d8fb89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:33.474591 17444 raft_consensus.cc:729] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:33.474704 17444 consensus_queue.cc:260] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:33.474802 12896 catalog_manager.cc:4830] TS 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109): tablet a746bcb2d718433f8c9acc8bd027a429 (table rebalance_test_table_11 [id=ba5c23cc430441c0a236a4359195fcb0]) successfully deleted
I20240503 13:56:33.474967 17444 ts_tablet_manager.cc:1398] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:33.475523 13423 tablet_service.cc:1467] Processing CreateTablet for tablet f48d92757c8a4b8d99910c5863d8fb89 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=46a3f42046eb4cb89a40f991a54af62b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:33.475759 13423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f48d92757c8a4b8d99910c5863d8fb89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:33.477880 15604 tablet_service.cc:1467] Processing CreateTablet for tablet 3f5d536cffe742e69f4b0d3b93caf76c (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=46a3f42046eb4cb89a40f991a54af62b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:33.477880 15605 tablet_service.cc:1467] Processing CreateTablet for tablet f48d92757c8a4b8d99910c5863d8fb89 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=46a3f42046eb4cb89a40f991a54af62b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:33.478773 15604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f5d536cffe742e69f4b0d3b93caf76c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:33.479313 12903 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalance_test_table_18" } modify_external_catalogs: true
I20240503 13:56:33.479434 12903 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalance_test_table_18" } modify_external_catalogs: true
I20240503 13:56:33.480027 17567 tablet_bootstrap.cc:492] T 3f5d536cffe742e69f4b0d3b93caf76c P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:33.480111 17567 ts_tablet_manager.cc:1367] T 3f5d536cffe742e69f4b0d3b93caf76c P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240503 13:56:33.480624 17567 raft_consensus.cc:348] T 3f5d536cffe742e69f4b0d3b93caf76c P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:33.480727 17567 raft_consensus.cc:374] T 3f5d536cffe742e69f4b0d3b93caf76c P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:33.480759 17567 raft_consensus.cc:729] T 3f5d536cffe742e69f4b0d3b93caf76c P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:33.480870 17567 consensus_queue.cc:260] T 3f5d536cffe742e69f4b0d3b93caf76c P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:33.481020 15605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f48d92757c8a4b8d99910c5863d8fb89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:33.481132 17661 tablet_bootstrap.cc:492] T 3f5d536cffe742e69f4b0d3b93caf76c P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:33.481187 17567 ts_tablet_manager.cc:1398] T 3f5d536cffe742e69f4b0d3b93caf76c P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:33.482014 17661 tablet_bootstrap.cc:654] T 3f5d536cffe742e69f4b0d3b93caf76c P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:33.482393 17567 tablet_bootstrap.cc:492] T cbd6d38ea18b484d80f4a918148efd3a P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:33.483036 17661 tablet_bootstrap.cc:492] T 3f5d536cffe742e69f4b0d3b93caf76c P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:33.483130 17661 ts_tablet_manager.cc:1367] T 3f5d536cffe742e69f4b0d3b93caf76c P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:56:33.483279 12903 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Sending DeleteTablet for 3 replicas of tablet 760e11f3264f4bf5b1865f1c2c46cd59
I20240503 13:56:33.483547 12903 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Sending DeleteTablet for 4 replicas of tablet 81d47f705c0b4128a84e4bf7d274a659
I20240503 13:56:33.483567 17661 raft_consensus.cc:348] T 3f5d536cffe742e69f4b0d3b93caf76c P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:33.483673 17661 raft_consensus.cc:374] T 3f5d536cffe742e69f4b0d3b93caf76c P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:33.483703 17661 raft_consensus.cc:729] T 3f5d536cffe742e69f4b0d3b93caf76c P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:33.483824 12903 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Sending DeleteTablet for 3 replicas of tablet 4bd42569fd3b4b39b0bd489acebbad10
I20240503 13:56:33.483816 17661 consensus_queue.cc:260] T 3f5d536cffe742e69f4b0d3b93caf76c P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:33.483915 13423 tablet_service.cc:1514] Processing DeleteTablet for tablet 760e11f3264f4bf5b1865f1c2c46cd59 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:33 UTC) from {username='slave'} at 127.0.0.1:33678
I20240503 13:56:33.484079 17661 ts_tablet_manager.cc:1398] T 3f5d536cffe742e69f4b0d3b93caf76c P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:33.484071 13424 tablet_service.cc:1514] Processing DeleteTablet for tablet 81d47f705c0b4128a84e4bf7d274a659 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:33 UTC) from {username='slave'} at 127.0.0.1:33678
I20240503 13:56:33.484169 17660 tablet_bootstrap.cc:492] T f48d92757c8a4b8d99910c5863d8fb89 P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:33.484232 17661 tablet_bootstrap.cc:492] T f48d92757c8a4b8d99910c5863d8fb89 P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:33.484246 16087 tablet_replica.cc:330] T 760e11f3264f4bf5b1865f1c2c46cd59 P 7690ee79577c45e3a2c8f8b552c0bff8: stopping tablet replica
I20240503 13:56:33.484355 16087 raft_consensus.cc:2229] T 760e11f3264f4bf5b1865f1c2c46cd59 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:33.484380 15867 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bd42569fd3b4b39b0bd489acebbad10 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:33 UTC) from {username='slave'} at 127.0.0.1:50498
I20240503 13:56:33.484478 16087 raft_consensus.cc:2258] T 760e11f3264f4bf5b1865f1c2c46cd59 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:33.485033 16087 ts_tablet_manager.cc:1875] T 760e11f3264f4bf5b1865f1c2c46cd59 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:33.485138 17661 tablet_bootstrap.cc:654] T f48d92757c8a4b8d99910c5863d8fb89 P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:33.485415 17660 tablet_bootstrap.cc:654] T f48d92757c8a4b8d99910c5863d8fb89 P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:33.486495 17660 tablet_bootstrap.cc:492] T f48d92757c8a4b8d99910c5863d8fb89 P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:33.486554 17660 ts_tablet_manager.cc:1367] T f48d92757c8a4b8d99910c5863d8fb89 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:33.487166 16087 ts_tablet_manager.cc:1888] T 760e11f3264f4bf5b1865f1c2c46cd59 P 7690ee79577c45e3a2c8f8b552c0bff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:56:33.487233 16087 log.cc:1196] T 760e11f3264f4bf5b1865f1c2c46cd59 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal/wals/760e11f3264f4bf5b1865f1c2c46cd59
I20240503 13:56:33.487267 17661 tablet_bootstrap.cc:492] T f48d92757c8a4b8d99910c5863d8fb89 P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:33.487334 17661 ts_tablet_manager.cc:1367] T f48d92757c8a4b8d99910c5863d8fb89 P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:56:33.486905 17660 raft_consensus.cc:348] T f48d92757c8a4b8d99910c5863d8fb89 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:33.487424 17660 raft_consensus.cc:374] T f48d92757c8a4b8d99910c5863d8fb89 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:33.487459 17660 raft_consensus.cc:729] T f48d92757c8a4b8d99910c5863d8fb89 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:33.487598 17660 consensus_queue.cc:260] T f48d92757c8a4b8d99910c5863d8fb89 P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:33.487874 17661 raft_consensus.cc:348] T f48d92757c8a4b8d99910c5863d8fb89 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:33.487967 17661 raft_consensus.cc:374] T f48d92757c8a4b8d99910c5863d8fb89 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:33.487995 17661 raft_consensus.cc:729] T f48d92757c8a4b8d99910c5863d8fb89 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:33.488116 17660 ts_tablet_manager.cc:1398] T f48d92757c8a4b8d99910c5863d8fb89 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:33.488126 17661 consensus_queue.cc:260] T f48d92757c8a4b8d99910c5863d8fb89 P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:33.488373 13033 tablet_service.cc:1514] Processing DeleteTablet for tablet 760e11f3264f4bf5b1865f1c2c46cd59 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:33 UTC) from {username='slave'} at 127.0.0.1:59458
I20240503 13:56:33.488564 13032 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bd42569fd3b4b39b0bd489acebbad10 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:33 UTC) from {username='slave'} at 127.0.0.1:59458
I20240503 13:56:33.488567 13034 tablet_service.cc:1514] Processing DeleteTablet for tablet 81d47f705c0b4128a84e4bf7d274a659 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:33 UTC) from {username='slave'} at 127.0.0.1:59458
I20240503 13:56:33.488751 16085 tablet_replica.cc:330] T 760e11f3264f4bf5b1865f1c2c46cd59 P 60ba3ffe3b7746469834cd91c84be14e: stopping tablet replica
I20240503 13:56:33.488826 13295 tablet_service.cc:1514] Processing DeleteTablet for tablet 81d47f705c0b4128a84e4bf7d274a659 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:33 UTC) from {username='slave'} at 127.0.0.1:33250
I20240503 13:56:33.488847 17661 ts_tablet_manager.cc:1398] T f48d92757c8a4b8d99910c5863d8fb89 P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:33.488873 16085 raft_consensus.cc:2229] T 760e11f3264f4bf5b1865f1c2c46cd59 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:33.489004 16085 raft_consensus.cc:2258] T 760e11f3264f4bf5b1865f1c2c46cd59 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:33.489012 17493 tablet_replica.cc:330] T 81d47f705c0b4128a84e4bf7d274a659 P ba0194098aa84449ae17e8a0d4c43703: stopping tablet replica
I20240503 13:56:33.489135 17493 raft_consensus.cc:2229] T 81d47f705c0b4128a84e4bf7d274a659 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:33.489171 13164 tablet_service.cc:1514] Processing DeleteTablet for tablet 760e11f3264f4bf5b1865f1c2c46cd59 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:33 UTC) from {username='slave'} at 127.0.0.1:41668
I20240503 13:56:33.489266 17493 raft_consensus.cc:2258] T 81d47f705c0b4128a84e4bf7d274a659 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:33.489333 16863 tablet_replica.cc:330] T 760e11f3264f4bf5b1865f1c2c46cd59 P 8ccf5a4ac450414c9d3579bd5800b770: stopping tablet replica
I20240503 13:56:33.489450 16863 raft_consensus.cc:2229] T 760e11f3264f4bf5b1865f1c2c46cd59 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:56:33.489495 16085 ts_tablet_manager.cc:1875] T 760e11f3264f4bf5b1865f1c2c46cd59 P 60ba3ffe3b7746469834cd91c84be14e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:33.489655 16863 raft_consensus.cc:2258] T 760e11f3264f4bf5b1865f1c2c46cd59 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:33.489840 17493 ts_tablet_manager.cc:1875] T 81d47f705c0b4128a84e4bf7d274a659 P ba0194098aa84449ae17e8a0d4c43703: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:33.490103 16863 ts_tablet_manager.cc:1875] T 760e11f3264f4bf5b1865f1c2c46cd59 P 8ccf5a4ac450414c9d3579bd5800b770: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:33.490885 16087 ts_tablet_manager.cc:1909] T 760e11f3264f4bf5b1865f1c2c46cd59 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting consensus metadata
I20240503 13:56:33.491647 16085 ts_tablet_manager.cc:1888] T 760e11f3264f4bf5b1865f1c2c46cd59 P 60ba3ffe3b7746469834cd91c84be14e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:56:33.491724 16085 log.cc:1196] T 760e11f3264f4bf5b1865f1c2c46cd59 P 60ba3ffe3b7746469834cd91c84be14e: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal/wals/760e11f3264f4bf5b1865f1c2c46cd59
I20240503 13:56:33.491983 16085 ts_tablet_manager.cc:1909] T 760e11f3264f4bf5b1865f1c2c46cd59 P 60ba3ffe3b7746469834cd91c84be14e: Deleting consensus metadata
I20240503 13:56:33.491997 17493 ts_tablet_manager.cc:1888] T 81d47f705c0b4128a84e4bf7d274a659 P ba0194098aa84449ae17e8a0d4c43703: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:33.492060 17493 log.cc:1196] T 81d47f705c0b4128a84e4bf7d274a659 P ba0194098aa84449ae17e8a0d4c43703: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/wal/wals/81d47f705c0b4128a84e4bf7d274a659
I20240503 13:56:33.492147 16087 tablet_replica.cc:330] T 81d47f705c0b4128a84e4bf7d274a659 P 7690ee79577c45e3a2c8f8b552c0bff8: stopping tablet replica
I20240503 13:56:33.492198 16863 ts_tablet_manager.cc:1888] T 760e11f3264f4bf5b1865f1c2c46cd59 P 8ccf5a4ac450414c9d3579bd5800b770: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 13:56:33.492257 16863 log.cc:1196] T 760e11f3264f4bf5b1865f1c2c46cd59 P 8ccf5a4ac450414c9d3579bd5800b770: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/wal/wals/760e11f3264f4bf5b1865f1c2c46cd59
I20240503 13:56:33.492259 16087 raft_consensus.cc:2229] T 81d47f705c0b4128a84e4bf7d274a659 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:56:33.492297 17493 ts_tablet_manager.cc:1909] T 81d47f705c0b4128a84e4bf7d274a659 P ba0194098aa84449ae17e8a0d4c43703: Deleting consensus metadata
I20240503 13:56:33.492565 13165 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bd42569fd3b4b39b0bd489acebbad10 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:33 UTC) from {username='slave'} at 127.0.0.1:41668
I20240503 13:56:33.492591 16863 ts_tablet_manager.cc:1909] T 760e11f3264f4bf5b1865f1c2c46cd59 P 8ccf5a4ac450414c9d3579bd5800b770: Deleting consensus metadata
I20240503 13:56:33.493186 15736 tablet_service.cc:1514] Processing DeleteTablet for tablet 81d47f705c0b4128a84e4bf7d274a659 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:33 UTC) from {username='slave'} at 127.0.0.1:44984
I20240503 13:56:33.493297 16863 tablet_replica.cc:330] T 4bd42569fd3b4b39b0bd489acebbad10 P 8ccf5a4ac450414c9d3579bd5800b770: stopping tablet replica
I20240503 13:56:33.493500 12899 catalog_manager.cc:4830] TS 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387): tablet 760e11f3264f4bf5b1865f1c2c46cd59 (table rebalance_test_table_18 [id=aa180ed819e24d89be0f5570e50d3c82]) successfully deleted
I20240503 13:56:33.493868 16087 raft_consensus.cc:2258] T 81d47f705c0b4128a84e4bf7d274a659 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:33.493968 17494 tablet_replica.cc:330] T 81d47f705c0b4128a84e4bf7d274a659 P 76c386bef61a4bd5a907266a5118ffe2: stopping tablet replica
I20240503 13:56:33.493949 12896 catalog_manager.cc:4830] TS 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109): tablet 760e11f3264f4bf5b1865f1c2c46cd59 (table rebalance_test_table_18 [id=aa180ed819e24d89be0f5570e50d3c82]) successfully deleted
I20240503 13:56:33.494091 17494 raft_consensus.cc:2229] T 81d47f705c0b4128a84e4bf7d274a659 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:33.494128 16863 raft_consensus.cc:2229] T 4bd42569fd3b4b39b0bd489acebbad10 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:33.494185 17494 raft_consensus.cc:2258] T 81d47f705c0b4128a84e4bf7d274a659 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:33.494284 16863 raft_consensus.cc:2258] T 4bd42569fd3b4b39b0bd489acebbad10 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:33.494365 16087 ts_tablet_manager.cc:1875] T 81d47f705c0b4128a84e4bf7d274a659 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:33.494637 17494 ts_tablet_manager.cc:1875] T 81d47f705c0b4128a84e4bf7d274a659 P 76c386bef61a4bd5a907266a5118ffe2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:33.494942 16085 tablet_replica.cc:330] T 4bd42569fd3b4b39b0bd489acebbad10 P 60ba3ffe3b7746469834cd91c84be14e: stopping tablet replica
I20240503 13:56:33.495038 16863 ts_tablet_manager.cc:1875] T 4bd42569fd3b4b39b0bd489acebbad10 P 8ccf5a4ac450414c9d3579bd5800b770: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:33.495026 12897 catalog_manager.cc:4830] TS 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695): tablet 760e11f3264f4bf5b1865f1c2c46cd59 (table rebalance_test_table_18 [id=aa180ed819e24d89be0f5570e50d3c82]) successfully deleted
I20240503 13:56:33.495055 16085 raft_consensus.cc:2229] T 4bd42569fd3b4b39b0bd489acebbad10 P 60ba3ffe3b7746469834cd91c84be14e [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:56:33.495247 16085 raft_consensus.cc:2258] T 4bd42569fd3b4b39b0bd489acebbad10 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:33.496656 16863 ts_tablet_manager.cc:1888] T 4bd42569fd3b4b39b0bd489acebbad10 P 8ccf5a4ac450414c9d3579bd5800b770: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240503 13:56:33.496713 16863 log.cc:1196] T 4bd42569fd3b4b39b0bd489acebbad10 P 8ccf5a4ac450414c9d3579bd5800b770: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/wal/wals/4bd42569fd3b4b39b0bd489acebbad10
I20240503 13:56:33.496934 16863 ts_tablet_manager.cc:1909] T 4bd42569fd3b4b39b0bd489acebbad10 P 8ccf5a4ac450414c9d3579bd5800b770: Deleting consensus metadata
I20240503 13:56:33.497344 17455 raft_consensus.cc:482] T f48d92757c8a4b8d99910c5863d8fb89 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:33.497510 17567 tablet_bootstrap.cc:654] T cbd6d38ea18b484d80f4a918148efd3a P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:33.497435 17455 raft_consensus.cc:504] T f48d92757c8a4b8d99910c5863d8fb89 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:33.497902 12899 catalog_manager.cc:4830] TS 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387): tablet 4bd42569fd3b4b39b0bd489acebbad10 (table rebalance_test_table_18 [id=aa180ed819e24d89be0f5570e50d3c82]) successfully deleted
I20240503 13:56:33.498066 17455 leader_election.cc:288] T f48d92757c8a4b8d99910c5863d8fb89 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885)
I20240503 13:56:33.498203 16087 ts_tablet_manager.cc:1888] T 81d47f705c0b4128a84e4bf7d274a659 P 7690ee79577c45e3a2c8f8b552c0bff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:33.498248 17663 tablet_replica.cc:330] T 4bd42569fd3b4b39b0bd489acebbad10 P e47105a0f46e4511b7112e571688a4c7: stopping tablet replica
I20240503 13:56:33.498262 16087 log.cc:1196] T 81d47f705c0b4128a84e4bf7d274a659 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal/wals/81d47f705c0b4128a84e4bf7d274a659
I20240503 13:56:33.498356 17663 raft_consensus.cc:2229] T 4bd42569fd3b4b39b0bd489acebbad10 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:33.498454 17663 raft_consensus.cc:2258] T 4bd42569fd3b4b39b0bd489acebbad10 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:33.498495 16087 ts_tablet_manager.cc:1909] T 81d47f705c0b4128a84e4bf7d274a659 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting consensus metadata
I20240503 13:56:33.498535 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f48d92757c8a4b8d99910c5863d8fb89" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2" is_pre_election: true
I20240503 13:56:33.498749 17658 tablet_bootstrap.cc:492] T cbd6d38ea18b484d80f4a918148efd3a P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
W20240503 13:56:33.498973 15561 leader_election.cc:341] T f48d92757c8a4b8d99910c5863d8fb89 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885): Illegal state: must be running to vote when last-logged opid is not known
I20240503 13:56:33.499104 17663 ts_tablet_manager.cc:1875] T 4bd42569fd3b4b39b0bd489acebbad10 P e47105a0f46e4511b7112e571688a4c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:33.499264 12896 catalog_manager.cc:4830] TS 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109): tablet 81d47f705c0b4128a84e4bf7d274a659 (table rebalance_test_table_18 [id=aa180ed819e24d89be0f5570e50d3c82]) successfully deleted
I20240503 13:56:33.499382 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f48d92757c8a4b8d99910c5863d8fb89" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
I20240503 13:56:33.499495 13447 raft_consensus.cc:2454] T f48d92757c8a4b8d99910c5863d8fb89 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 0.
I20240503 13:56:33.499696 17658 tablet_bootstrap.cc:654] T cbd6d38ea18b484d80f4a918148efd3a P 61b887f0d40d497badeca5604c1f0546: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:33.499783 15558 leader_election.cc:302] T f48d92757c8a4b8d99910c5863d8fb89 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 7690ee79577c45e3a2c8f8b552c0bff8; no voters: 76c386bef61a4bd5a907266a5118ffe2
I20240503 13:56:33.500031 17455 raft_consensus.cc:2781] T f48d92757c8a4b8d99910c5863d8fb89 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:33.500075 17455 raft_consensus.cc:482] T f48d92757c8a4b8d99910c5863d8fb89 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:33.500108 17455 raft_consensus.cc:3037] T f48d92757c8a4b8d99910c5863d8fb89 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:33.500761 17494 ts_tablet_manager.cc:1888] T 81d47f705c0b4128a84e4bf7d274a659 P 76c386bef61a4bd5a907266a5118ffe2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:33.500831 17494 log.cc:1196] T 81d47f705c0b4128a84e4bf7d274a659 P 76c386bef61a4bd5a907266a5118ffe2: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/wal/wals/81d47f705c0b4128a84e4bf7d274a659
I20240503 13:56:33.501097 17494 ts_tablet_manager.cc:1909] T 81d47f705c0b4128a84e4bf7d274a659 P 76c386bef61a4bd5a907266a5118ffe2: Deleting consensus metadata
I20240503 13:56:33.501077 17455 raft_consensus.cc:504] T f48d92757c8a4b8d99910c5863d8fb89 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:33.501238 17663 ts_tablet_manager.cc:1888] T 4bd42569fd3b4b39b0bd489acebbad10 P e47105a0f46e4511b7112e571688a4c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240503 13:56:33.501302 17663 log.cc:1196] T 4bd42569fd3b4b39b0bd489acebbad10 P e47105a0f46e4511b7112e571688a4c7: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-7/wal/wals/4bd42569fd3b4b39b0bd489acebbad10
I20240503 13:56:33.501425 17455 leader_election.cc:288] T f48d92757c8a4b8d99910c5863d8fb89 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 election: Requested vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885)
I20240503 13:56:33.501533 17663 ts_tablet_manager.cc:1909] T 4bd42569fd3b4b39b0bd489acebbad10 P e47105a0f46e4511b7112e571688a4c7: Deleting consensus metadata
I20240503 13:56:33.501871 17654 tablet_bootstrap.cc:492] T 3f5d536cffe742e69f4b0d3b93caf76c P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:33.501886 12899 catalog_manager.cc:4830] TS 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885): tablet 81d47f705c0b4128a84e4bf7d274a659 (table rebalance_test_table_18 [id=aa180ed819e24d89be0f5570e50d3c82]) successfully deleted
I20240503 13:56:33.501972 17654 ts_tablet_manager.cc:1367] T 3f5d536cffe742e69f4b0d3b93caf76c P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.000s
I20240503 13:56:33.502245 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f48d92757c8a4b8d99910c5863d8fb89" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2"
I20240503 13:56:33.502429 17654 raft_consensus.cc:348] T 3f5d536cffe742e69f4b0d3b93caf76c P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:33.502504 12899 catalog_manager.cc:4830] TS ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279): tablet 81d47f705c0b4128a84e4bf7d274a659 (table rebalance_test_table_18 [id=aa180ed819e24d89be0f5570e50d3c82]) successfully deleted
I20240503 13:56:33.502549 17654 raft_consensus.cc:374] T 3f5d536cffe742e69f4b0d3b93caf76c P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:33.502585 17654 raft_consensus.cc:729] T 3f5d536cffe742e69f4b0d3b93caf76c P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
W20240503 13:56:33.502683 15561 leader_election.cc:341] T f48d92757c8a4b8d99910c5863d8fb89 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885): Illegal state: must be running to vote when last-logged opid is not known
I20240503 13:56:33.502707 17654 consensus_queue.cc:260] T 3f5d536cffe742e69f4b0d3b93caf76c P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:33.502979 17654 ts_tablet_manager.cc:1398] T 3f5d536cffe742e69f4b0d3b93caf76c P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:33.503039 12896 catalog_manager.cc:4830] TS e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): tablet 4bd42569fd3b4b39b0bd489acebbad10 (table rebalance_test_table_18 [id=aa180ed819e24d89be0f5570e50d3c82]) successfully deleted
I20240503 13:56:33.503166 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f48d92757c8a4b8d99910c5863d8fb89" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
I20240503 13:56:33.503263 13447 raft_consensus.cc:3037] T f48d92757c8a4b8d99910c5863d8fb89 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:33.504207 13447 raft_consensus.cc:2454] T f48d92757c8a4b8d99910c5863d8fb89 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 1.
I20240503 13:56:33.504735 15558 leader_election.cc:302] T f48d92757c8a4b8d99910c5863d8fb89 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 7690ee79577c45e3a2c8f8b552c0bff8; no voters: 76c386bef61a4bd5a907266a5118ffe2
I20240503 13:56:33.505365 17455 raft_consensus.cc:2781] T f48d92757c8a4b8d99910c5863d8fb89 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:33.505549 17455 raft_consensus.cc:686] T f48d92757c8a4b8d99910c5863d8fb89 P 009f849ee7b54c80a3f5264695047e4d [term 1 LEADER]: Becoming Leader. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Running, Role: LEADER
I20240503 13:56:33.505681 17455 consensus_queue.cc:237] T f48d92757c8a4b8d99910c5863d8fb89 P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:33.505884 16085 ts_tablet_manager.cc:1875] T 4bd42569fd3b4b39b0bd489acebbad10 P 60ba3ffe3b7746469834cd91c84be14e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:33.507776 12907 catalog_manager.cc:5484] T f48d92757c8a4b8d99910c5863d8fb89 P 009f849ee7b54c80a3f5264695047e4d reported cstate change: term changed from 0 to 1, leader changed from <none> to 009f849ee7b54c80a3f5264695047e4d (127.6.98.70). New cstate: current_term: 1 leader_uuid: "009f849ee7b54c80a3f5264695047e4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:33.508378 16085 ts_tablet_manager.cc:1888] T 4bd42569fd3b4b39b0bd489acebbad10 P 60ba3ffe3b7746469834cd91c84be14e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240503 13:56:33.508461 16085 log.cc:1196] T 4bd42569fd3b4b39b0bd489acebbad10 P 60ba3ffe3b7746469834cd91c84be14e: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal/wals/4bd42569fd3b4b39b0bd489acebbad10
I20240503 13:56:33.508738 16085 ts_tablet_manager.cc:1909] T 4bd42569fd3b4b39b0bd489acebbad10 P 60ba3ffe3b7746469834cd91c84be14e: Deleting consensus metadata
I20240503 13:56:33.509722 16085 tablet_replica.cc:330] T 81d47f705c0b4128a84e4bf7d274a659 P 60ba3ffe3b7746469834cd91c84be14e: stopping tablet replica
I20240503 13:56:33.509804 12897 catalog_manager.cc:4830] TS 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695): tablet 4bd42569fd3b4b39b0bd489acebbad10 (table rebalance_test_table_18 [id=aa180ed819e24d89be0f5570e50d3c82]) successfully deleted
I20240503 13:56:33.509951 16085 raft_consensus.cc:2229] T 81d47f705c0b4128a84e4bf7d274a659 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:33.510113 16085 raft_consensus.cc:2258] T 81d47f705c0b4128a84e4bf7d274a659 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:33.510644 16085 ts_tablet_manager.cc:1875] T 81d47f705c0b4128a84e4bf7d274a659 P 60ba3ffe3b7746469834cd91c84be14e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:33.512686 16085 ts_tablet_manager.cc:1888] T 81d47f705c0b4128a84e4bf7d274a659 P 60ba3ffe3b7746469834cd91c84be14e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:33.512816 16085 log.cc:1196] T 81d47f705c0b4128a84e4bf7d274a659 P 60ba3ffe3b7746469834cd91c84be14e: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal/wals/81d47f705c0b4128a84e4bf7d274a659
I20240503 13:56:33.513141 16085 ts_tablet_manager.cc:1909] T 81d47f705c0b4128a84e4bf7d274a659 P 60ba3ffe3b7746469834cd91c84be14e: Deleting consensus metadata
I20240503 13:56:33.513692 12897 catalog_manager.cc:4830] TS 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695): tablet 81d47f705c0b4128a84e4bf7d274a659 (table rebalance_test_table_18 [id=aa180ed819e24d89be0f5570e50d3c82]) successfully deleted
I20240503 13:56:33.514212 17567 tablet_bootstrap.cc:492] T cbd6d38ea18b484d80f4a918148efd3a P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:33.514318 17567 ts_tablet_manager.cc:1367] T cbd6d38ea18b484d80f4a918148efd3a P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.000s
I20240503 13:56:33.514510 17658 tablet_bootstrap.cc:492] T cbd6d38ea18b484d80f4a918148efd3a P 61b887f0d40d497badeca5604c1f0546: No bootstrap required, opened a new log
I20240503 13:56:33.514616 17658 ts_tablet_manager.cc:1367] T cbd6d38ea18b484d80f4a918148efd3a P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.002s
I20240503 13:56:33.514891 17567 raft_consensus.cc:348] T cbd6d38ea18b484d80f4a918148efd3a P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:33.514997 17567 raft_consensus.cc:374] T cbd6d38ea18b484d80f4a918148efd3a P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:33.515027 17567 raft_consensus.cc:729] T cbd6d38ea18b484d80f4a918148efd3a P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:33.515074 17658 raft_consensus.cc:348] T cbd6d38ea18b484d80f4a918148efd3a P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:33.515175 17658 raft_consensus.cc:374] T cbd6d38ea18b484d80f4a918148efd3a P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:33.515156 17567 consensus_queue.cc:260] T cbd6d38ea18b484d80f4a918148efd3a P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:33.515229 17658 raft_consensus.cc:729] T cbd6d38ea18b484d80f4a918148efd3a P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:33.515347 17658 consensus_queue.cc:260] T cbd6d38ea18b484d80f4a918148efd3a P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:33.515417 17567 ts_tablet_manager.cc:1398] T cbd6d38ea18b484d80f4a918148efd3a P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:33.515580 17567 tablet_bootstrap.cc:492] T f48d92757c8a4b8d99910c5863d8fb89 P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:33.515607 17658 ts_tablet_manager.cc:1398] T cbd6d38ea18b484d80f4a918148efd3a P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:33.516453 17567 tablet_bootstrap.cc:654] T f48d92757c8a4b8d99910c5863d8fb89 P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:33.518716 17567 tablet_bootstrap.cc:492] T f48d92757c8a4b8d99910c5863d8fb89 P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:33.518811 17567 ts_tablet_manager.cc:1367] T f48d92757c8a4b8d99910c5863d8fb89 P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:56:33.519315 17567 raft_consensus.cc:348] T f48d92757c8a4b8d99910c5863d8fb89 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:33.519448 17567 raft_consensus.cc:374] T f48d92757c8a4b8d99910c5863d8fb89 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:33.519529 17567 raft_consensus.cc:729] T f48d92757c8a4b8d99910c5863d8fb89 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:33.519716 17567 consensus_queue.cc:260] T f48d92757c8a4b8d99910c5863d8fb89 P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:33.520071 17567 ts_tablet_manager.cc:1398] T f48d92757c8a4b8d99910c5863d8fb89 P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:33.568442 17350 raft_consensus.cc:482] T f48d92757c8a4b8d99910c5863d8fb89 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:33.568596 17350 raft_consensus.cc:504] T f48d92757c8a4b8d99910c5863d8fb89 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:33.568972 17350 leader_election.cc:288] T f48d92757c8a4b8d99910c5863d8fb89 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951), 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:33.569450 15624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f48d92757c8a4b8d99910c5863d8fb89" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d" is_pre_election: true
I20240503 13:56:33.569979 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f48d92757c8a4b8d99910c5863d8fb89" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
I20240503 13:56:33.570158 13447 raft_consensus.cc:2379] T f48d92757c8a4b8d99910c5863d8fb89 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 76c386bef61a4bd5a907266a5118ffe2 in current term 1: Already voted for candidate 009f849ee7b54c80a3f5264695047e4d in this term.
I20240503 13:56:33.571125 15691 leader_election.cc:302] T f48d92757c8a4b8d99910c5863d8fb89 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76c386bef61a4bd5a907266a5118ffe2; no voters: 009f849ee7b54c80a3f5264695047e4d, 7690ee79577c45e3a2c8f8b552c0bff8
I20240503 13:56:33.571339 17350 raft_consensus.cc:3037] T f48d92757c8a4b8d99910c5863d8fb89 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:33.572445 17350 raft_consensus.cc:2726] T f48d92757c8a4b8d99910c5863d8fb89 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 13:56:33.580117 12903 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:56:33.585924 17645 tablet.cc:1722] T 9f973b16bcb04791ba29d5e3f4fe37a6 P 76c386bef61a4bd5a907266a5118ffe2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:33.586267 17645 tablet.cc:1620] T 9f973b16bcb04791ba29d5e3f4fe37a6 P 76c386bef61a4bd5a907266a5118ffe2: MemRowSet was empty: no flush needed.
I20240503 13:56:33.587097 17648 tablet.cc:1722] T 9f973b16bcb04791ba29d5e3f4fe37a6 P 8ccf5a4ac450414c9d3579bd5800b770: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:33.587399 17648 tablet.cc:1620] T 9f973b16bcb04791ba29d5e3f4fe37a6 P 8ccf5a4ac450414c9d3579bd5800b770: MemRowSet was empty: no flush needed.
I20240503 13:56:33.588352 17646 tablet.cc:1722] T 9f973b16bcb04791ba29d5e3f4fe37a6 P e47105a0f46e4511b7112e571688a4c7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:33.588663 17646 tablet.cc:1620] T 9f973b16bcb04791ba29d5e3f4fe37a6 P e47105a0f46e4511b7112e571688a4c7: MemRowSet was empty: no flush needed.
I20240503 13:56:33.593806 12909 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=b043b5f53d944ef7822ca72165b0099e] alter complete (version 2)
I20240503 13:56:33.608633 17668 raft_consensus.cc:482] T 3f5d536cffe742e69f4b0d3b93caf76c P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:33.608762 17668 raft_consensus.cc:504] T 3f5d536cffe742e69f4b0d3b93caf76c P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:33.609216 17668 leader_election.cc:288] T 3f5d536cffe742e69f4b0d3b93caf76c P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885), 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:33.609669 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f5d536cffe742e69f4b0d3b93caf76c" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2" is_pre_election: true
I20240503 13:56:33.609830 15757 raft_consensus.cc:2454] T 3f5d536cffe742e69f4b0d3b93caf76c P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 0.
I20240503 13:56:33.610121 15561 leader_election.cc:302] T 3f5d536cffe742e69f4b0d3b93caf76c P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 76c386bef61a4bd5a907266a5118ffe2; no voters:
I20240503 13:56:33.610292 17668 raft_consensus.cc:2781] T 3f5d536cffe742e69f4b0d3b93caf76c P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:33.610359 17668 raft_consensus.cc:482] T 3f5d536cffe742e69f4b0d3b93caf76c P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:33.610397 17668 raft_consensus.cc:3037] T 3f5d536cffe742e69f4b0d3b93caf76c P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:33.611346 17668 raft_consensus.cc:504] T 3f5d536cffe742e69f4b0d3b93caf76c P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:33.611716 17668 leader_election.cc:288] T 3f5d536cffe742e69f4b0d3b93caf76c P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 election: Requested vote from peers 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885), 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:33.612000 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f5d536cffe742e69f4b0d3b93caf76c" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2"
I20240503 13:56:33.612108 15757 raft_consensus.cc:3037] T 3f5d536cffe742e69f4b0d3b93caf76c P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:33.612952 15757 raft_consensus.cc:2454] T 3f5d536cffe742e69f4b0d3b93caf76c P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 1.
I20240503 13:56:33.613224 15561 leader_election.cc:302] T 3f5d536cffe742e69f4b0d3b93caf76c P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 76c386bef61a4bd5a907266a5118ffe2; no voters:
I20240503 13:56:33.613272 13051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f5d536cffe742e69f4b0d3b93caf76c" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e" is_pre_election: true
I20240503 13:56:33.613384 17668 raft_consensus.cc:2781] T 3f5d536cffe742e69f4b0d3b93caf76c P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:33.613346 13052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f5d536cffe742e69f4b0d3b93caf76c" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e"
I20240503 13:56:33.613464 17668 raft_consensus.cc:686] T 3f5d536cffe742e69f4b0d3b93caf76c P 009f849ee7b54c80a3f5264695047e4d [term 1 LEADER]: Becoming Leader. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Running, Role: LEADER
I20240503 13:56:33.613519 13051 raft_consensus.cc:2454] T 3f5d536cffe742e69f4b0d3b93caf76c P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 0.
I20240503 13:56:33.613600 17668 consensus_queue.cc:237] T 3f5d536cffe742e69f4b0d3b93caf76c P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:33.615643 12907 catalog_manager.cc:5484] T 3f5d536cffe742e69f4b0d3b93caf76c P 009f849ee7b54c80a3f5264695047e4d reported cstate change: term changed from 0 to 1, leader changed from <none> to 009f849ee7b54c80a3f5264695047e4d (127.6.98.70). New cstate: current_term: 1 leader_uuid: "009f849ee7b54c80a3f5264695047e4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } }
I20240503 13:56:33.678262 17659 raft_consensus.cc:482] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:33.678390 17659 raft_consensus.cc:504] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:33.678848 17659 leader_election.cc:288] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885), 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153)
I20240503 13:56:33.679679 15493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbd6d38ea18b484d80f4a918148efd3a" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546" is_pre_election: true
I20240503 13:56:33.679845 15493 raft_consensus.cc:2454] T cbd6d38ea18b484d80f4a918148efd3a P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 0.
I20240503 13:56:33.680189 13121 leader_election.cc:302] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61b887f0d40d497badeca5604c1f0546, 8ccf5a4ac450414c9d3579bd5800b770; no voters:
I20240503 13:56:33.680349 17659 raft_consensus.cc:2781] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:33.680399 17659 raft_consensus.cc:482] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:33.680446 17659 raft_consensus.cc:3037] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:33.681357 17659 raft_consensus.cc:504] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:33.681689 17659 leader_election.cc:288] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 election: Requested vote from peers 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885), 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153)
I20240503 13:56:33.682018 15493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbd6d38ea18b484d80f4a918148efd3a" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546"
I20240503 13:56:33.682159 15493 raft_consensus.cc:3037] T cbd6d38ea18b484d80f4a918148efd3a P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:33.682688 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbd6d38ea18b484d80f4a918148efd3a" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2" is_pre_election: true
I20240503 13:56:33.682742 15758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbd6d38ea18b484d80f4a918148efd3a" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2"
I20240503 13:56:33.682804 15757 raft_consensus.cc:2454] T cbd6d38ea18b484d80f4a918148efd3a P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 0.
I20240503 13:56:33.683147 15493 raft_consensus.cc:2454] T cbd6d38ea18b484d80f4a918148efd3a P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 1.
I20240503 13:56:33.683367 13121 leader_election.cc:302] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61b887f0d40d497badeca5604c1f0546, 8ccf5a4ac450414c9d3579bd5800b770; no voters: 76c386bef61a4bd5a907266a5118ffe2
I20240503 13:56:33.683506 17659 raft_consensus.cc:2781] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:33.683581 17659 raft_consensus.cc:686] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 LEADER]: Becoming Leader. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Running, Role: LEADER
I20240503 13:56:33.683712 17659 consensus_queue.cc:237] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:33.685647 12907 catalog_manager.cc:5484] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66). New cstate: current_term: 1 leader_uuid: "8ccf5a4ac450414c9d3579bd5800b770" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } }
I20240503 13:56:33.695401 12907 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:56:33.703310 17645 tablet.cc:1722] T 9f973b16bcb04791ba29d5e3f4fe37a6 P 76c386bef61a4bd5a907266a5118ffe2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:33.703848 17648 tablet.cc:1722] T 9f973b16bcb04791ba29d5e3f4fe37a6 P 8ccf5a4ac450414c9d3579bd5800b770: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:33.704237 17646 tablet.cc:1722] T 9f973b16bcb04791ba29d5e3f4fe37a6 P e47105a0f46e4511b7112e571688a4c7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:33.709964 12903 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table_new_name [id=b043b5f53d944ef7822ca72165b0099e] alter complete (version 3)
I20240503 13:56:33.712675 15605 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:56:33.716338 13034 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:33.725179 15472 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:56:33.734298 13424 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:56:33.736202 15736 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:56:33.736305 13295 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 13:56:33.743556 13165 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:33.744616 15867 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:56:33.812281 13314 consensus_queue.cc:237] T 1b714fbd920a447f866b8d4547275675 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:33.812753 12909 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:33.812891 12909 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:33.813879 13447 raft_consensus.cc:1261] T 1b714fbd920a447f866b8d4547275675 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:56:33.814157 15758 raft_consensus.cc:1261] T 1b714fbd920a447f866b8d4547275675 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:56:33.814282 17200 consensus_queue.cc:1035] T 1b714fbd920a447f866b8d4547275675 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:33.814553 16890 consensus_queue.cc:1035] T 1b714fbd920a447f866b8d4547275675 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:33.815778 12909 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Sending DeleteTablet for 3 replicas of tablet 9f973b16bcb04791ba29d5e3f4fe37a6
I20240503 13:56:33.816571 15736 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f973b16bcb04791ba29d5e3f4fe37a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:33 UTC) from {username='slave'} at 127.0.0.1:44984
I20240503 13:56:33.816572 15867 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f973b16bcb04791ba29d5e3f4fe37a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:33 UTC) from {username='slave'} at 127.0.0.1:50498
I20240503 13:56:33.816761 17663 tablet_replica.cc:330] T 9f973b16bcb04791ba29d5e3f4fe37a6 P e47105a0f46e4511b7112e571688a4c7: stopping tablet replica
I20240503 13:56:33.816766 13165 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f973b16bcb04791ba29d5e3f4fe37a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:33 UTC) from {username='slave'} at 127.0.0.1:41668
I20240503 13:56:33.816926 16863 tablet_replica.cc:330] T 9f973b16bcb04791ba29d5e3f4fe37a6 P 8ccf5a4ac450414c9d3579bd5800b770: stopping tablet replica
I20240503 13:56:33.816915 17663 raft_consensus.cc:2229] T 9f973b16bcb04791ba29d5e3f4fe37a6 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:33.817047 17663 raft_consensus.cc:2258] T 9f973b16bcb04791ba29d5e3f4fe37a6 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:33.817057 16863 raft_consensus.cc:2229] T 9f973b16bcb04791ba29d5e3f4fe37a6 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:33.817011 16890 raft_consensus.cc:2932] T 1b714fbd920a447f866b8d4547275675 P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:33.817164 16863 raft_consensus.cc:2258] T 9f973b16bcb04791ba29d5e3f4fe37a6 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:33.817225 17494 tablet_replica.cc:330] T 9f973b16bcb04791ba29d5e3f4fe37a6 P 76c386bef61a4bd5a907266a5118ffe2: stopping tablet replica
I20240503 13:56:33.817358 17494 raft_consensus.cc:2229] T 9f973b16bcb04791ba29d5e3f4fe37a6 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:56:33.817559 17494 raft_consensus.cc:2258] T 9f973b16bcb04791ba29d5e3f4fe37a6 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:33.817756 16863 ts_tablet_manager.cc:1875] T 9f973b16bcb04791ba29d5e3f4fe37a6 P 8ccf5a4ac450414c9d3579bd5800b770: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:33.818104 17494 ts_tablet_manager.cc:1875] T 9f973b16bcb04791ba29d5e3f4fe37a6 P 76c386bef61a4bd5a907266a5118ffe2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:33.819495 16863 ts_tablet_manager.cc:1888] T 9f973b16bcb04791ba29d5e3f4fe37a6 P 8ccf5a4ac450414c9d3579bd5800b770: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:33.819567 16863 log.cc:1196] T 9f973b16bcb04791ba29d5e3f4fe37a6 P 8ccf5a4ac450414c9d3579bd5800b770: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/wal/wals/9f973b16bcb04791ba29d5e3f4fe37a6
I20240503 13:56:33.819555 15758 raft_consensus.cc:2932] T 1b714fbd920a447f866b8d4547275675 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:33.819801 16863 ts_tablet_manager.cc:1909] T 9f973b16bcb04791ba29d5e3f4fe37a6 P 8ccf5a4ac450414c9d3579bd5800b770: Deleting consensus metadata
I20240503 13:56:33.820179 17494 ts_tablet_manager.cc:1888] T 9f973b16bcb04791ba29d5e3f4fe37a6 P 76c386bef61a4bd5a907266a5118ffe2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:33.820241 17494 log.cc:1196] T 9f973b16bcb04791ba29d5e3f4fe37a6 P 76c386bef61a4bd5a907266a5118ffe2: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/wal/wals/9f973b16bcb04791ba29d5e3f4fe37a6
I20240503 13:56:33.820108 13447 raft_consensus.cc:2932] T 1b714fbd920a447f866b8d4547275675 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:33.820468 17494 ts_tablet_manager.cc:1909] T 9f973b16bcb04791ba29d5e3f4fe37a6 P 76c386bef61a4bd5a907266a5118ffe2: Deleting consensus metadata
I20240503 13:56:33.820742 12909 catalog_manager.cc:5484] T 1b714fbd920a447f866b8d4547275675 P ba0194098aa84449ae17e8a0d4c43703 reported cstate change: config changed from index 4 to 5, VOTER 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66) evicted. New cstate: current_term: 1 leader_uuid: "ba0194098aa84449ae17e8a0d4c43703" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:33.821635 17663 ts_tablet_manager.cc:1875] T 9f973b16bcb04791ba29d5e3f4fe37a6 P e47105a0f46e4511b7112e571688a4c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:33.823393 12899 catalog_manager.cc:4830] TS 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885): tablet 9f973b16bcb04791ba29d5e3f4fe37a6 (table rebalancer_dynamic_table_new_name [id=b043b5f53d944ef7822ca72165b0099e]) successfully deleted
I20240503 13:56:33.823707 17663 ts_tablet_manager.cc:1888] T 9f973b16bcb04791ba29d5e3f4fe37a6 P e47105a0f46e4511b7112e571688a4c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:33.823794 17663 log.cc:1196] T 9f973b16bcb04791ba29d5e3f4fe37a6 P e47105a0f46e4511b7112e571688a4c7: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-7/wal/wals/9f973b16bcb04791ba29d5e3f4fe37a6
I20240503 13:56:33.824026 17663 ts_tablet_manager.cc:1909] T 9f973b16bcb04791ba29d5e3f4fe37a6 P e47105a0f46e4511b7112e571688a4c7: Deleting consensus metadata
I20240503 13:56:33.824384 12899 catalog_manager.cc:4830] TS 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387): tablet 9f973b16bcb04791ba29d5e3f4fe37a6 (table rebalancer_dynamic_table_new_name [id=b043b5f53d944ef7822ca72165b0099e]) successfully deleted
I20240503 13:56:33.825264 13165 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b714fbd920a447f866b8d4547275675 with delete_type TABLET_DATA_TOMBSTONED (TS 8ccf5a4ac450414c9d3579bd5800b770 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:41668
I20240503 13:56:33.825461 16863 tablet_replica.cc:330] T 1b714fbd920a447f866b8d4547275675 P 8ccf5a4ac450414c9d3579bd5800b770: stopping tablet replica
I20240503 13:56:33.825589 16863 raft_consensus.cc:2229] T 1b714fbd920a447f866b8d4547275675 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:33.825588 12896 catalog_manager.cc:4830] TS e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): tablet 9f973b16bcb04791ba29d5e3f4fe37a6 (table rebalancer_dynamic_table_new_name [id=b043b5f53d944ef7822ca72165b0099e]) successfully deleted
I20240503 13:56:33.825734 16863 raft_consensus.cc:2258] T 1b714fbd920a447f866b8d4547275675 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:33.826225 16863 ts_tablet_manager.cc:1875] T 1b714fbd920a447f866b8d4547275675 P 8ccf5a4ac450414c9d3579bd5800b770: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:56:33.829598 16863 ts_tablet_manager.cc:1888] T 1b714fbd920a447f866b8d4547275675 P 8ccf5a4ac450414c9d3579bd5800b770: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 13:56:33.829725 16863 log.cc:1196] T 1b714fbd920a447f866b8d4547275675 P 8ccf5a4ac450414c9d3579bd5800b770: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/wal/wals/1b714fbd920a447f866b8d4547275675
I20240503 13:56:33.830189 12899 catalog_manager.cc:4830] TS 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387): tablet 1b714fbd920a447f866b8d4547275675 (table rebalance_test_table_13 [id=4d62f84fd7bc47e8a1f2de384d64b787]) successfully deleted
I20240503 13:56:33.900468 15472 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:56:33.902258 15736 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:56:33.905041 13165 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:33.905411 13034 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:33.906199 13295 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 13:56:33.909507 15605 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:56:33.917857 12909 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:56:33.921505 13424 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:56:33.925820 13424 tablet_service.cc:1467] Processing CreateTablet for tablet e8f12042b09947b6812f2d70ec1176e0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=cf0f26e8e19d47e09ce29d5ec3ab1919]), partition=
I20240503 13:56:33.925818 15736 tablet_service.cc:1467] Processing CreateTablet for tablet e8f12042b09947b6812f2d70ec1176e0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=cf0f26e8e19d47e09ce29d5ec3ab1919]), partition=
I20240503 13:56:33.926074 15736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8f12042b09947b6812f2d70ec1176e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:33.926092 13424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8f12042b09947b6812f2d70ec1176e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:33.928251 17660 tablet_bootstrap.cc:492] T e8f12042b09947b6812f2d70ec1176e0 P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:33.928390 15605 tablet_service.cc:1467] Processing CreateTablet for tablet e8f12042b09947b6812f2d70ec1176e0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=cf0f26e8e19d47e09ce29d5ec3ab1919]), partition=
I20240503 13:56:33.928642 15605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8f12042b09947b6812f2d70ec1176e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:33.929124 17660 tablet_bootstrap.cc:654] T e8f12042b09947b6812f2d70ec1176e0 P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:33.929747 17567 tablet_bootstrap.cc:492] T e8f12042b09947b6812f2d70ec1176e0 P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:33.930526 17567 tablet_bootstrap.cc:654] T e8f12042b09947b6812f2d70ec1176e0 P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:33.931069 17661 tablet_bootstrap.cc:492] T e8f12042b09947b6812f2d70ec1176e0 P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:33.931842 17661 tablet_bootstrap.cc:654] T e8f12042b09947b6812f2d70ec1176e0 P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:33.950726 15867 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:56:33.955610 17661 tablet_bootstrap.cc:492] T e8f12042b09947b6812f2d70ec1176e0 P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:33.955722 17661 ts_tablet_manager.cc:1367] T e8f12042b09947b6812f2d70ec1176e0 P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.000s
I20240503 13:56:33.956203 17661 raft_consensus.cc:348] T e8f12042b09947b6812f2d70ec1176e0 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:33.956318 17661 raft_consensus.cc:374] T e8f12042b09947b6812f2d70ec1176e0 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:33.956346 17661 raft_consensus.cc:729] T e8f12042b09947b6812f2d70ec1176e0 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:33.956452 17661 consensus_queue.cc:260] T e8f12042b09947b6812f2d70ec1176e0 P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:33.956760 17661 ts_tablet_manager.cc:1398] T e8f12042b09947b6812f2d70ec1176e0 P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:33.961889 17668 consensus_queue.cc:1035] T f48d92757c8a4b8d99910c5863d8fb89 P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:33.963568 17567 tablet_bootstrap.cc:492] T e8f12042b09947b6812f2d70ec1176e0 P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:33.963670 17567 ts_tablet_manager.cc:1367] T e8f12042b09947b6812f2d70ec1176e0 P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.034s user 0.002s sys 0.000s
I20240503 13:56:33.964098 17567 raft_consensus.cc:348] T e8f12042b09947b6812f2d70ec1176e0 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:33.964195 17567 raft_consensus.cc:374] T e8f12042b09947b6812f2d70ec1176e0 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:33.964221 17567 raft_consensus.cc:729] T e8f12042b09947b6812f2d70ec1176e0 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:33.964330 17567 consensus_queue.cc:260] T e8f12042b09947b6812f2d70ec1176e0 P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:33.965261 17567 ts_tablet_manager.cc:1398] T e8f12042b09947b6812f2d70ec1176e0 P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:33.966254 17660 tablet_bootstrap.cc:492] T e8f12042b09947b6812f2d70ec1176e0 P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:33.966344 17660 ts_tablet_manager.cc:1367] T e8f12042b09947b6812f2d70ec1176e0 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.038s user 0.002s sys 0.000s
I20240503 13:56:33.966774 17660 raft_consensus.cc:348] T e8f12042b09947b6812f2d70ec1176e0 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:33.966873 17660 raft_consensus.cc:374] T e8f12042b09947b6812f2d70ec1176e0 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:33.966899 17660 raft_consensus.cc:729] T e8f12042b09947b6812f2d70ec1176e0 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:33.967016 17660 consensus_queue.cc:260] T e8f12042b09947b6812f2d70ec1176e0 P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:33.967259 17660 ts_tablet_manager.cc:1398] T e8f12042b09947b6812f2d70ec1176e0 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:33.976943 17668 consensus_queue.cc:1035] T f48d92757c8a4b8d99910c5863d8fb89 P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:33.988277 13316 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9eb0d911701b4b2bb33a6b7ab59f08aa"
dest_uuid: "ba0194098aa84449ae17e8a0d4c43703"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58488
I20240503 13:56:33.988420 13316 raft_consensus.cc:595] T 9eb0d911701b4b2bb33a6b7ab59f08aa P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Received request to transfer leadership
I20240503 13:56:34.040445 17455 raft_consensus.cc:482] T e8f12042b09947b6812f2d70ec1176e0 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:34.040616 17455 raft_consensus.cc:504] T e8f12042b09947b6812f2d70ec1176e0 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:34.041086 17455 leader_election.cc:288] T e8f12042b09947b6812f2d70ec1176e0 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885)
I20240503 13:56:34.041587 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8f12042b09947b6812f2d70ec1176e0" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
I20240503 13:56:34.041729 13447 raft_consensus.cc:2454] T e8f12042b09947b6812f2d70ec1176e0 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 0.
I20240503 13:56:34.041688 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8f12042b09947b6812f2d70ec1176e0" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2" is_pre_election: true
I20240503 13:56:34.041860 15757 raft_consensus.cc:2454] T e8f12042b09947b6812f2d70ec1176e0 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 0.
I20240503 13:56:34.042184 15558 leader_election.cc:302] T e8f12042b09947b6812f2d70ec1176e0 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 7690ee79577c45e3a2c8f8b552c0bff8; no voters:
I20240503 13:56:34.042649 17455 raft_consensus.cc:2781] T e8f12042b09947b6812f2d70ec1176e0 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:34.042706 17455 raft_consensus.cc:482] T e8f12042b09947b6812f2d70ec1176e0 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:34.042752 17455 raft_consensus.cc:3037] T e8f12042b09947b6812f2d70ec1176e0 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:34.043637 17455 raft_consensus.cc:504] T e8f12042b09947b6812f2d70ec1176e0 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:34.043941 17455 leader_election.cc:288] T e8f12042b09947b6812f2d70ec1176e0 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 election: Requested vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885)
I20240503 13:56:34.044238 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8f12042b09947b6812f2d70ec1176e0" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
I20240503 13:56:34.044267 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8f12042b09947b6812f2d70ec1176e0" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2"
I20240503 13:56:34.044355 13447 raft_consensus.cc:3037] T e8f12042b09947b6812f2d70ec1176e0 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:34.044379 15757 raft_consensus.cc:3037] T e8f12042b09947b6812f2d70ec1176e0 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:34.045332 15757 raft_consensus.cc:2454] T e8f12042b09947b6812f2d70ec1176e0 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 1.
I20240503 13:56:34.045558 13447 raft_consensus.cc:2454] T e8f12042b09947b6812f2d70ec1176e0 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 1.
I20240503 13:56:34.045789 15561 leader_election.cc:302] T e8f12042b09947b6812f2d70ec1176e0 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 76c386bef61a4bd5a907266a5118ffe2; no voters:
I20240503 13:56:34.046329 17455 raft_consensus.cc:2781] T e8f12042b09947b6812f2d70ec1176e0 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:34.046415 17455 raft_consensus.cc:686] T e8f12042b09947b6812f2d70ec1176e0 P 009f849ee7b54c80a3f5264695047e4d [term 1 LEADER]: Becoming Leader. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Running, Role: LEADER
I20240503 13:56:34.046545 17455 consensus_queue.cc:237] T e8f12042b09947b6812f2d70ec1176e0 P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:34.048918 12903 catalog_manager.cc:5484] T e8f12042b09947b6812f2d70ec1176e0 P 009f849ee7b54c80a3f5264695047e4d reported cstate change: term changed from 0 to 1, leader changed from <none> to 009f849ee7b54c80a3f5264695047e4d (127.6.98.70). New cstate: current_term: 1 leader_uuid: "009f849ee7b54c80a3f5264695047e4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:34.068213 13051 raft_consensus.cc:3037] T 3f5d536cffe742e69f4b0d3b93caf76c P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:34.071197 17455 consensus_queue.cc:1035] T 3f5d536cffe742e69f4b0d3b93caf76c P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:34.072460 15605 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:56:34.075968 13034 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:34.078725 17668 consensus_queue.cc:1035] T 3f5d536cffe742e69f4b0d3b93caf76c P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:34.083832 15472 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:56:34.083853 15736 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:56:34.093582 13295 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 13:56:34.097086 13424 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:56:34.108665 15867 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:56:34.110240 13165 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:34.131563 15757 raft_consensus.cc:3037] T cbd6d38ea18b484d80f4a918148efd3a P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:34.133236 17659 consensus_queue.cc:1035] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:34.140686 16949 consensus_queue.cc:1035] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:34.162279 12908 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:56:34.166340 15757 raft_consensus.cc:1261] T e8f12042b09947b6812f2d70ec1176e0 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Refusing update from remote peer 009f849ee7b54c80a3f5264695047e4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:34.166699 17668 consensus_queue.cc:1035] T e8f12042b09947b6812f2d70ec1176e0 P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:34.168107 13447 raft_consensus.cc:1261] T e8f12042b09947b6812f2d70ec1176e0 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer 009f849ee7b54c80a3f5264695047e4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:34.168602 17668 consensus_queue.cc:1035] T e8f12042b09947b6812f2d70ec1176e0 P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:34.168807 12908 catalog_manager.cc:5576] TS 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885) does not have the latest schema for tablet e8f12042b09947b6812f2d70ec1176e0 (table rebalancer_dynamic_table [id=cf0f26e8e19d47e09ce29d5ec3ab1919]). Expected version 1 got 0
I20240503 13:56:34.171182 12908 catalog_manager.cc:5576] TS 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109) does not have the latest schema for tablet e8f12042b09947b6812f2d70ec1176e0 (table rebalancer_dynamic_table [id=cf0f26e8e19d47e09ce29d5ec3ab1919]). Expected version 1 got 0
I20240503 13:56:34.171986 17873 tablet.cc:1722] T e8f12042b09947b6812f2d70ec1176e0 P 7690ee79577c45e3a2c8f8b552c0bff8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:34.172333 17873 tablet.cc:1620] T e8f12042b09947b6812f2d70ec1176e0 P 7690ee79577c45e3a2c8f8b552c0bff8: MemRowSet was empty: no flush needed.
I20240503 13:56:34.172477 17645 tablet.cc:1722] T e8f12042b09947b6812f2d70ec1176e0 P 76c386bef61a4bd5a907266a5118ffe2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:34.172785 17645 tablet.cc:1620] T e8f12042b09947b6812f2d70ec1176e0 P 76c386bef61a4bd5a907266a5118ffe2: MemRowSet was empty: no flush needed.
I20240503 13:56:34.176643 17872 tablet.cc:1722] T e8f12042b09947b6812f2d70ec1176e0 P 009f849ee7b54c80a3f5264695047e4d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:34.176908 17872 tablet.cc:1620] T e8f12042b09947b6812f2d70ec1176e0 P 009f849ee7b54c80a3f5264695047e4d: MemRowSet was empty: no flush needed.
I20240503 13:56:34.178062 12903 catalog_manager.cc:5576] TS 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109) does not have the latest schema for tablet e8f12042b09947b6812f2d70ec1176e0 (table rebalancer_dynamic_table [id=cf0f26e8e19d47e09ce29d5ec3ab1919]). Expected version 1 got 0
I20240503 13:56:34.178532 12908 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=cf0f26e8e19d47e09ce29d5ec3ab1919] alter complete (version 1)
I20240503 13:56:34.221160 17200 raft_consensus.cc:979] T 9eb0d911701b4b2bb33a6b7ab59f08aa P ba0194098aa84449ae17e8a0d4c43703: : Instructing follower 61b887f0d40d497badeca5604c1f0546 to start an election
I20240503 13:56:34.221283 17200 raft_consensus.cc:1067] T 9eb0d911701b4b2bb33a6b7ab59f08aa P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Signalling peer 61b887f0d40d497badeca5604c1f0546 to start an election
I20240503 13:56:34.222573 15493 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9eb0d911701b4b2bb33a6b7ab59f08aa"
dest_uuid: "61b887f0d40d497badeca5604c1f0546"
from {username='slave'} at 127.6.98.67:53759
I20240503 13:56:34.222746 15493 raft_consensus.cc:482] T 9eb0d911701b4b2bb33a6b7ab59f08aa P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 13:56:34.222807 15493 raft_consensus.cc:3037] T 9eb0d911701b4b2bb33a6b7ab59f08aa P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:56:34.223917 15493 raft_consensus.cc:504] T 9eb0d911701b4b2bb33a6b7ab59f08aa P 61b887f0d40d497badeca5604c1f0546 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:34.224424 15493 leader_election.cc:288] T 9eb0d911701b4b2bb33a6b7ab59f08aa P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 2 election: Requested vote from peers 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387), 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:34.225255 13314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eb0d911701b4b2bb33a6b7ab59f08aa" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "ba0194098aa84449ae17e8a0d4c43703"
I20240503 13:56:34.225373 13314 raft_consensus.cc:3032] T 9eb0d911701b4b2bb33a6b7ab59f08aa P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 13:56:34.225421 13314 raft_consensus.cc:729] T 9eb0d911701b4b2bb33a6b7ab59f08aa P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Running, Role: LEADER
I20240503 13:56:34.225553 13314 consensus_queue.cc:260] T 9eb0d911701b4b2bb33a6b7ab59f08aa P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:34.225735 13314 raft_consensus.cc:3037] T 9eb0d911701b4b2bb33a6b7ab59f08aa P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:56:34.225858 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eb0d911701b4b2bb33a6b7ab59f08aa" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
I20240503 13:56:34.225968 13447 raft_consensus.cc:3037] T 9eb0d911701b4b2bb33a6b7ab59f08aa P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:56:34.226775 13314 raft_consensus.cc:2454] T 9eb0d911701b4b2bb33a6b7ab59f08aa P ba0194098aa84449ae17e8a0d4c43703 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 2.
I20240503 13:56:34.226852 13447 raft_consensus.cc:2454] T 9eb0d911701b4b2bb33a6b7ab59f08aa P 7690ee79577c45e3a2c8f8b552c0bff8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 2.
I20240503 13:56:34.227414 15429 leader_election.cc:302] T 9eb0d911701b4b2bb33a6b7ab59f08aa P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 61b887f0d40d497badeca5604c1f0546, 7690ee79577c45e3a2c8f8b552c0bff8, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:34.227588 17858 raft_consensus.cc:2781] T 9eb0d911701b4b2bb33a6b7ab59f08aa P 61b887f0d40d497badeca5604c1f0546 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:56:34.227663 17858 raft_consensus.cc:686] T 9eb0d911701b4b2bb33a6b7ab59f08aa P 61b887f0d40d497badeca5604c1f0546 [term 2 LEADER]: Becoming Leader. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Running, Role: LEADER
I20240503 13:56:34.227789 17858 consensus_queue.cc:237] T 9eb0d911701b4b2bb33a6b7ab59f08aa P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:34.229784 12907 catalog_manager.cc:5484] T 9eb0d911701b4b2bb33a6b7ab59f08aa P 61b887f0d40d497badeca5604c1f0546 reported cstate change: term changed from 1 to 2, leader changed from ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) to 61b887f0d40d497badeca5604c1f0546 (127.6.98.69). New cstate: current_term: 2 leader_uuid: "61b887f0d40d497badeca5604c1f0546" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:34.231040 12906 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_extra_table_14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:56:34.236936 15867 tablet_service.cc:1467] Processing CreateTablet for tablet 4b4b26adad184e1ea57c00a44635679c (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=b6ca31bb3860435ba9c230a2fd41d931]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:34.236996 15736 tablet_service.cc:1467] Processing CreateTablet for tablet 4b4b26adad184e1ea57c00a44635679c (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=b6ca31bb3860435ba9c230a2fd41d931]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:34.237251 15867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b4b26adad184e1ea57c00a44635679c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:34.237263 15736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b4b26adad184e1ea57c00a44635679c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:34.239367 13034 tablet_service.cc:1467] Processing CreateTablet for tablet f58fc3bc579748bb88de34612808ccda (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=b6ca31bb3860435ba9c230a2fd41d931]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:34.239470 13424 tablet_service.cc:1467] Processing CreateTablet for tablet 4b4b26adad184e1ea57c00a44635679c (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=b6ca31bb3860435ba9c230a2fd41d931]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:34.239629 13034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f58fc3bc579748bb88de34612808ccda. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:34.239715 13424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b4b26adad184e1ea57c00a44635679c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:34.241750 17567 tablet_bootstrap.cc:492] T 4b4b26adad184e1ea57c00a44635679c P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:34.242408 15867 tablet_service.cc:1467] Processing CreateTablet for tablet f58fc3bc579748bb88de34612808ccda (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=b6ca31bb3860435ba9c230a2fd41d931]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:34.242646 15867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f58fc3bc579748bb88de34612808ccda. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:34.242677 17567 tablet_bootstrap.cc:654] T 4b4b26adad184e1ea57c00a44635679c P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:34.243070 17904 tablet_bootstrap.cc:492] T 4b4b26adad184e1ea57c00a44635679c P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:34.243386 17660 tablet_bootstrap.cc:492] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:34.243846 13034 tablet_service.cc:1467] Processing CreateTablet for tablet 3c197f47582546e39cda80c63bc98f23 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=b6ca31bb3860435ba9c230a2fd41d931]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:34.244068 13034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c197f47582546e39cda80c63bc98f23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:34.244119 17904 tablet_bootstrap.cc:654] T 4b4b26adad184e1ea57c00a44635679c P e47105a0f46e4511b7112e571688a4c7: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:34.244167 17660 tablet_bootstrap.cc:654] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:34.244939 15605 tablet_service.cc:1467] Processing CreateTablet for tablet 3c197f47582546e39cda80c63bc98f23 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=b6ca31bb3860435ba9c230a2fd41d931]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:34.245173 15605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c197f47582546e39cda80c63bc98f23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:34.245376 17904 tablet_bootstrap.cc:492] T 4b4b26adad184e1ea57c00a44635679c P e47105a0f46e4511b7112e571688a4c7: No bootstrap required, opened a new log
I20240503 13:56:34.245455 17904 ts_tablet_manager.cc:1367] T 4b4b26adad184e1ea57c00a44635679c P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:56:34.245882 17904 raft_consensus.cc:348] T 4b4b26adad184e1ea57c00a44635679c P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:34.245983 17904 raft_consensus.cc:374] T 4b4b26adad184e1ea57c00a44635679c P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:34.246009 17904 raft_consensus.cc:729] T 4b4b26adad184e1ea57c00a44635679c P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:34.246124 17904 consensus_queue.cc:260] T 4b4b26adad184e1ea57c00a44635679c P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:34.246392 17904 ts_tablet_manager.cc:1398] T 4b4b26adad184e1ea57c00a44635679c P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:34.246548 17904 tablet_bootstrap.cc:492] T f58fc3bc579748bb88de34612808ccda P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:34.247331 17661 tablet_bootstrap.cc:492] T 3c197f47582546e39cda80c63bc98f23 P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:34.247413 17904 tablet_bootstrap.cc:654] T f58fc3bc579748bb88de34612808ccda P e47105a0f46e4511b7112e571688a4c7: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:34.249675 15472 tablet_service.cc:1467] Processing CreateTablet for tablet 3c197f47582546e39cda80c63bc98f23 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=b6ca31bb3860435ba9c230a2fd41d931]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:34.249935 15472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c197f47582546e39cda80c63bc98f23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:34.250512 17661 tablet_bootstrap.cc:654] T 3c197f47582546e39cda80c63bc98f23 P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:34.251422 13424 tablet_service.cc:1467] Processing CreateTablet for tablet f58fc3bc579748bb88de34612808ccda (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=b6ca31bb3860435ba9c230a2fd41d931]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:34.251668 13424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f58fc3bc579748bb88de34612808ccda. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:34.252084 17661 tablet_bootstrap.cc:492] T 3c197f47582546e39cda80c63bc98f23 P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:34.252161 17661 ts_tablet_manager.cc:1367] T 3c197f47582546e39cda80c63bc98f23 P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 13:56:34.252625 17661 raft_consensus.cc:348] T 3c197f47582546e39cda80c63bc98f23 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:34.252748 17661 raft_consensus.cc:374] T 3c197f47582546e39cda80c63bc98f23 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:34.252790 17661 raft_consensus.cc:729] T 3c197f47582546e39cda80c63bc98f23 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:34.252947 17660 tablet_bootstrap.cc:492] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:34.252904 17661 consensus_queue.cc:260] T 3c197f47582546e39cda80c63bc98f23 P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:34.253021 17660 ts_tablet_manager.cc:1367] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20240503 13:56:34.253202 17661 ts_tablet_manager.cc:1398] T 3c197f47582546e39cda80c63bc98f23 P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:34.255343 17904 tablet_bootstrap.cc:492] T f58fc3bc579748bb88de34612808ccda P e47105a0f46e4511b7112e571688a4c7: No bootstrap required, opened a new log
I20240503 13:56:34.255439 17904 ts_tablet_manager.cc:1367] T f58fc3bc579748bb88de34612808ccda P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 13:56:34.255857 17904 raft_consensus.cc:348] T f58fc3bc579748bb88de34612808ccda P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:34.255966 17904 raft_consensus.cc:374] T f58fc3bc579748bb88de34612808ccda P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:34.255998 17904 raft_consensus.cc:729] T f58fc3bc579748bb88de34612808ccda P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:34.256119 17904 consensus_queue.cc:260] T f58fc3bc579748bb88de34612808ccda P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:34.256374 17904 ts_tablet_manager.cc:1398] T f58fc3bc579748bb88de34612808ccda P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:34.256685 17660 raft_consensus.cc:348] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:34.256786 17660 raft_consensus.cc:374] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:34.256819 17660 raft_consensus.cc:729] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:34.256923 17660 consensus_queue.cc:260] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:34.257208 17660 ts_tablet_manager.cc:1398] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 13:56:34.257361 17660 tablet_bootstrap.cc:492] T f58fc3bc579748bb88de34612808ccda P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:34.261641 17660 tablet_bootstrap.cc:654] T f58fc3bc579748bb88de34612808ccda P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:34.263276 13424 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:56:34.264003 17567 tablet_bootstrap.cc:492] T 4b4b26adad184e1ea57c00a44635679c P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:34.264102 17567 ts_tablet_manager.cc:1367] T 4b4b26adad184e1ea57c00a44635679c P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20240503 13:56:34.264643 17567 raft_consensus.cc:348] T 4b4b26adad184e1ea57c00a44635679c P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:34.264770 17567 raft_consensus.cc:374] T 4b4b26adad184e1ea57c00a44635679c P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:34.264806 17567 raft_consensus.cc:729] T 4b4b26adad184e1ea57c00a44635679c P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:34.264930 17567 consensus_queue.cc:260] T 4b4b26adad184e1ea57c00a44635679c P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:34.265209 17567 ts_tablet_manager.cc:1398] T 4b4b26adad184e1ea57c00a44635679c P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:34.266211 13034 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:34.269204 17660 tablet_bootstrap.cc:492] T f58fc3bc579748bb88de34612808ccda P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:34.269305 17660 ts_tablet_manager.cc:1367] T f58fc3bc579748bb88de34612808ccda P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 13:56:34.269773 17660 raft_consensus.cc:348] T f58fc3bc579748bb88de34612808ccda P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:34.269872 17660 raft_consensus.cc:374] T f58fc3bc579748bb88de34612808ccda P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:34.269899 17660 raft_consensus.cc:729] T f58fc3bc579748bb88de34612808ccda P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:34.270843 17660 consensus_queue.cc:260] T f58fc3bc579748bb88de34612808ccda P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:34.271819 13185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eb0d911701b4b2bb33a6b7ab59f08aa" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770"
I20240503 13:56:34.271940 13185 raft_consensus.cc:3037] T 9eb0d911701b4b2bb33a6b7ab59f08aa P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:56:34.272282 17660 ts_tablet_manager.cc:1398] T f58fc3bc579748bb88de34612808ccda P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:56:34.272629 17911 tablet_bootstrap.cc:492] T 3c197f47582546e39cda80c63bc98f23 P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:34.272792 17905 tablet_bootstrap.cc:492] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:34.273001 13185 raft_consensus.cc:2454] T 9eb0d911701b4b2bb33a6b7ab59f08aa P 8ccf5a4ac450414c9d3579bd5800b770 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 2.
I20240503 13:56:34.273571 17911 tablet_bootstrap.cc:654] T 3c197f47582546e39cda80c63bc98f23 P 61b887f0d40d497badeca5604c1f0546: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:34.273619 17905 tablet_bootstrap.cc:654] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:34.274631 17911 tablet_bootstrap.cc:492] T 3c197f47582546e39cda80c63bc98f23 P 61b887f0d40d497badeca5604c1f0546: No bootstrap required, opened a new log
I20240503 13:56:34.274643 17905 tablet_bootstrap.cc:492] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:34.274710 17905 ts_tablet_manager.cc:1367] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:34.274710 17911 ts_tablet_manager.cc:1367] T 3c197f47582546e39cda80c63bc98f23 P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:56:34.275089 17911 raft_consensus.cc:348] T 3c197f47582546e39cda80c63bc98f23 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:34.275089 17905 raft_consensus.cc:348] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:34.275185 17905 raft_consensus.cc:374] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:34.275185 17911 raft_consensus.cc:374] T 3c197f47582546e39cda80c63bc98f23 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:34.275215 17905 raft_consensus.cc:729] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:34.275215 17911 raft_consensus.cc:729] T 3c197f47582546e39cda80c63bc98f23 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:34.275343 17911 consensus_queue.cc:260] T 3c197f47582546e39cda80c63bc98f23 P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:34.275338 17905 consensus_queue.cc:260] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:34.275602 17905 ts_tablet_manager.cc:1398] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:34.275602 17911 ts_tablet_manager.cc:1398] T 3c197f47582546e39cda80c63bc98f23 P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:34.275759 17905 tablet_bootstrap.cc:492] T 3c197f47582546e39cda80c63bc98f23 P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:34.276791 17905 tablet_bootstrap.cc:654] T 3c197f47582546e39cda80c63bc98f23 P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:34.280319 15605 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:56:34.280707 15472 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:56:34.280910 13295 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:56:34.282653 12909 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:56:34.289687 15867 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:56:34.297335 17872 tablet.cc:1722] T e8f12042b09947b6812f2d70ec1176e0 P 009f849ee7b54c80a3f5264695047e4d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:34.297678 17872 tablet.cc:1620] T e8f12042b09947b6812f2d70ec1176e0 P 009f849ee7b54c80a3f5264695047e4d: MemRowSet was empty: no flush needed.
I20240503 13:56:34.302497 17873 tablet.cc:1722] T e8f12042b09947b6812f2d70ec1176e0 P 7690ee79577c45e3a2c8f8b552c0bff8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:34.302815 17873 tablet.cc:1620] T e8f12042b09947b6812f2d70ec1176e0 P 7690ee79577c45e3a2c8f8b552c0bff8: MemRowSet was empty: no flush needed.
I20240503 13:56:34.303305 12909 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=cf0f26e8e19d47e09ce29d5ec3ab1919] alter complete (version 2)
I20240503 13:56:34.307471 13165 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:34.310710 17645 tablet.cc:1722] T e8f12042b09947b6812f2d70ec1176e0 P 76c386bef61a4bd5a907266a5118ffe2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:34.311031 17645 tablet.cc:1620] T e8f12042b09947b6812f2d70ec1176e0 P 76c386bef61a4bd5a907266a5118ffe2: MemRowSet was empty: no flush needed.
I20240503 13:56:34.314457 15736 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:56:34.343609 17905 tablet_bootstrap.cc:492] T 3c197f47582546e39cda80c63bc98f23 P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:34.343734 17905 ts_tablet_manager.cc:1367] T 3c197f47582546e39cda80c63bc98f23 P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.068s user 0.002s sys 0.000s
I20240503 13:56:34.344209 17905 raft_consensus.cc:348] T 3c197f47582546e39cda80c63bc98f23 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:34.344333 17905 raft_consensus.cc:374] T 3c197f47582546e39cda80c63bc98f23 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:34.344367 17905 raft_consensus.cc:729] T 3c197f47582546e39cda80c63bc98f23 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:34.344516 17905 consensus_queue.cc:260] T 3c197f47582546e39cda80c63bc98f23 P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:34.344821 17905 ts_tablet_manager.cc:1398] T 3c197f47582546e39cda80c63bc98f23 P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:34.369912 13314 consensus_queue.cc:237] T 3bb148fcf60f410089420a9499f08086 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:34.371099 13447 raft_consensus.cc:1261] T 3bb148fcf60f410089420a9499f08086 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:56:34.371250 15493 raft_consensus.cc:1261] T 3bb148fcf60f410089420a9499f08086 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:56:34.371709 17555 consensus_queue.cc:1035] T 3bb148fcf60f410089420a9499f08086 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:34.371881 16890 consensus_queue.cc:1035] T 3bb148fcf60f410089420a9499f08086 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:34.373786 17555 raft_consensus.cc:2932] T 3bb148fcf60f410089420a9499f08086 P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:34.376077 15493 raft_consensus.cc:2932] T 3bb148fcf60f410089420a9499f08086 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:34.376399 12907 catalog_manager.cc:5484] T 3bb148fcf60f410089420a9499f08086 P ba0194098aa84449ae17e8a0d4c43703 reported cstate change: config changed from index 4 to 5, VOTER 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66) evicted. New cstate: current_term: 1 leader_uuid: "ba0194098aa84449ae17e8a0d4c43703" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:34.380390 16404 raft_consensus.cc:482] T f58fc3bc579748bb88de34612808ccda P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:34.380527 16404 raft_consensus.cc:504] T f58fc3bc579748bb88de34612808ccda P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:34.380728 13165 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bb148fcf60f410089420a9499f08086 with delete_type TABLET_DATA_TOMBSTONED (TS 8ccf5a4ac450414c9d3579bd5800b770 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:41668
I20240503 13:56:34.380995 16404 leader_election.cc:288] T f58fc3bc579748bb88de34612808ccda P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695), 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:34.381052 17951 tablet_replica.cc:330] T 3bb148fcf60f410089420a9499f08086 P 8ccf5a4ac450414c9d3579bd5800b770: stopping tablet replica
I20240503 13:56:34.381182 17951 raft_consensus.cc:2229] T 3bb148fcf60f410089420a9499f08086 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:34.381448 17951 raft_consensus.cc:2258] T 3bb148fcf60f410089420a9499f08086 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:34.381459 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f58fc3bc579748bb88de34612808ccda" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
I20240503 13:56:34.381600 13445 raft_consensus.cc:2454] T f58fc3bc579748bb88de34612808ccda P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 0.
I20240503 13:56:34.381886 13052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f58fc3bc579748bb88de34612808ccda" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e" is_pre_election: true
I20240503 13:56:34.382028 13052 raft_consensus.cc:2454] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 0.
I20240503 13:56:34.382052 17951 ts_tablet_manager.cc:1875] T 3bb148fcf60f410089420a9499f08086 P 8ccf5a4ac450414c9d3579bd5800b770: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:56:34.382426 15822 leader_election.cc:302] T f58fc3bc579748bb88de34612808ccda P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:34.382612 16404 raft_consensus.cc:2781] T f58fc3bc579748bb88de34612808ccda P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:34.382648 16404 raft_consensus.cc:482] T f58fc3bc579748bb88de34612808ccda P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:34.382673 16404 raft_consensus.cc:3037] T f58fc3bc579748bb88de34612808ccda P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:34.383316 16404 raft_consensus.cc:504] T f58fc3bc579748bb88de34612808ccda P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:34.383584 16404 leader_election.cc:288] T f58fc3bc579748bb88de34612808ccda P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 election: Requested vote from peers 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695), 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:34.383936 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f58fc3bc579748bb88de34612808ccda" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
I20240503 13:56:34.384060 13445 raft_consensus.cc:3037] T f58fc3bc579748bb88de34612808ccda P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:34.384303 17951 ts_tablet_manager.cc:1888] T 3bb148fcf60f410089420a9499f08086 P 8ccf5a4ac450414c9d3579bd5800b770: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 13:56:34.384362 17951 log.cc:1196] T 3bb148fcf60f410089420a9499f08086 P 8ccf5a4ac450414c9d3579bd5800b770: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/wal/wals/3bb148fcf60f410089420a9499f08086
I20240503 13:56:34.384966 12899 catalog_manager.cc:4830] TS 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387): tablet 3bb148fcf60f410089420a9499f08086 (table rebalance_test_table_17 [id=52f6b247496e4c3eb63eef4f16160012]) successfully deleted
I20240503 13:56:34.385040 13445 raft_consensus.cc:2454] T f58fc3bc579748bb88de34612808ccda P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 1.
I20240503 13:56:34.385331 15822 leader_election.cc:302] T f58fc3bc579748bb88de34612808ccda P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:34.385682 13051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f58fc3bc579748bb88de34612808ccda" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e"
I20240503 13:56:34.385790 13051 raft_consensus.cc:3037] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:34.386200 16404 raft_consensus.cc:2781] T f58fc3bc579748bb88de34612808ccda P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:34.386336 16404 raft_consensus.cc:686] T f58fc3bc579748bb88de34612808ccda P e47105a0f46e4511b7112e571688a4c7 [term 1 LEADER]: Becoming Leader. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Running, Role: LEADER
I20240503 13:56:34.386452 16404 consensus_queue.cc:237] T f58fc3bc579748bb88de34612808ccda P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:34.386710 13051 raft_consensus.cc:2454] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 1.
I20240503 13:56:34.388108 12909 catalog_manager.cc:5484] T f58fc3bc579748bb88de34612808ccda P e47105a0f46e4511b7112e571688a4c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e47105a0f46e4511b7112e571688a4c7 (127.6.98.72). New cstate: current_term: 1 leader_uuid: "e47105a0f46e4511b7112e571688a4c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:34.389762 13447 raft_consensus.cc:2932] T 3bb148fcf60f410089420a9499f08086 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:34.415562 12909 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:56:34.421582 17872 tablet.cc:1722] T e8f12042b09947b6812f2d70ec1176e0 P 009f849ee7b54c80a3f5264695047e4d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:34.421931 17645 tablet.cc:1722] T e8f12042b09947b6812f2d70ec1176e0 P 76c386bef61a4bd5a907266a5118ffe2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:34.426537 17873 tablet.cc:1722] T e8f12042b09947b6812f2d70ec1176e0 P 7690ee79577c45e3a2c8f8b552c0bff8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:34.427743 12909 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table_new_name [id=cf0f26e8e19d47e09ce29d5ec3ab1919] alter complete (version 3)
I20240503 13:56:34.516198 17953 raft_consensus.cc:482] T 4b4b26adad184e1ea57c00a44635679c P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:34.516311 17953 raft_consensus.cc:504] T 4b4b26adad184e1ea57c00a44635679c P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:34.516685 17953 leader_election.cc:288] T 4b4b26adad184e1ea57c00a44635679c P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885), 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:34.517058 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b4b26adad184e1ea57c00a44635679c" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
I20240503 13:56:34.517095 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b4b26adad184e1ea57c00a44635679c" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2" is_pre_election: true
I20240503 13:56:34.517216 15757 raft_consensus.cc:2454] T 4b4b26adad184e1ea57c00a44635679c P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 0.
I20240503 13:56:34.517216 13445 raft_consensus.cc:2454] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 0.
I20240503 13:56:34.517519 15825 leader_election.cc:302] T 4b4b26adad184e1ea57c00a44635679c P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76c386bef61a4bd5a907266a5118ffe2, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:34.517678 17953 raft_consensus.cc:2781] T 4b4b26adad184e1ea57c00a44635679c P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:34.517731 17953 raft_consensus.cc:482] T 4b4b26adad184e1ea57c00a44635679c P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:34.517756 17953 raft_consensus.cc:3037] T 4b4b26adad184e1ea57c00a44635679c P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:34.518628 17953 raft_consensus.cc:504] T 4b4b26adad184e1ea57c00a44635679c P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:34.518958 17953 leader_election.cc:288] T 4b4b26adad184e1ea57c00a44635679c P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 election: Requested vote from peers 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885), 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:34.519280 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b4b26adad184e1ea57c00a44635679c" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2"
I20240503 13:56:34.519320 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b4b26adad184e1ea57c00a44635679c" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
I20240503 13:56:34.519404 15757 raft_consensus.cc:3037] T 4b4b26adad184e1ea57c00a44635679c P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:34.519443 13445 raft_consensus.cc:3037] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:34.520373 15757 raft_consensus.cc:2454] T 4b4b26adad184e1ea57c00a44635679c P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 1.
I20240503 13:56:34.520373 13445 raft_consensus.cc:2454] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 1.
I20240503 13:56:34.520696 15822 leader_election.cc:302] T 4b4b26adad184e1ea57c00a44635679c P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:34.520864 17953 raft_consensus.cc:2781] T 4b4b26adad184e1ea57c00a44635679c P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:34.521235 17953 raft_consensus.cc:686] T 4b4b26adad184e1ea57c00a44635679c P e47105a0f46e4511b7112e571688a4c7 [term 1 LEADER]: Becoming Leader. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Running, Role: LEADER
I20240503 13:56:34.521395 17953 consensus_queue.cc:237] T 4b4b26adad184e1ea57c00a44635679c P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:34.523550 12909 catalog_manager.cc:5484] T 4b4b26adad184e1ea57c00a44635679c P e47105a0f46e4511b7112e571688a4c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e47105a0f46e4511b7112e571688a4c7 (127.6.98.72). New cstate: current_term: 1 leader_uuid: "e47105a0f46e4511b7112e571688a4c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } }
I20240503 13:56:34.534168 12909 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:34.534401 12909 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:34.536545 12909 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Sending DeleteTablet for 3 replicas of tablet e8f12042b09947b6812f2d70ec1176e0
I20240503 13:56:34.537112 15736 tablet_service.cc:1514] Processing DeleteTablet for tablet e8f12042b09947b6812f2d70ec1176e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:34 UTC) from {username='slave'} at 127.0.0.1:44984
I20240503 13:56:34.537197 15605 tablet_service.cc:1514] Processing DeleteTablet for tablet e8f12042b09947b6812f2d70ec1176e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:34 UTC) from {username='slave'} at 127.0.0.1:60718
I20240503 13:56:34.537197 13424 tablet_service.cc:1514] Processing DeleteTablet for tablet e8f12042b09947b6812f2d70ec1176e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:34 UTC) from {username='slave'} at 127.0.0.1:33678
I20240503 13:56:34.537529 17987 tablet_replica.cc:330] T e8f12042b09947b6812f2d70ec1176e0 P 009f849ee7b54c80a3f5264695047e4d: stopping tablet replica
I20240503 13:56:34.537597 17986 tablet_replica.cc:330] T e8f12042b09947b6812f2d70ec1176e0 P 76c386bef61a4bd5a907266a5118ffe2: stopping tablet replica
I20240503 13:56:34.537678 17987 raft_consensus.cc:2229] T e8f12042b09947b6812f2d70ec1176e0 P 009f849ee7b54c80a3f5264695047e4d [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:56:34.537712 17986 raft_consensus.cc:2229] T e8f12042b09947b6812f2d70ec1176e0 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:34.537825 17986 raft_consensus.cc:2258] T e8f12042b09947b6812f2d70ec1176e0 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:34.537983 17987 raft_consensus.cc:2258] T e8f12042b09947b6812f2d70ec1176e0 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:34.538319 17988 tablet_replica.cc:330] T e8f12042b09947b6812f2d70ec1176e0 P 7690ee79577c45e3a2c8f8b552c0bff8: stopping tablet replica
I20240503 13:56:34.538419 17986 ts_tablet_manager.cc:1875] T e8f12042b09947b6812f2d70ec1176e0 P 76c386bef61a4bd5a907266a5118ffe2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:34.538451 17988 raft_consensus.cc:2229] T e8f12042b09947b6812f2d70ec1176e0 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:34.538594 17987 ts_tablet_manager.cc:1875] T e8f12042b09947b6812f2d70ec1176e0 P 009f849ee7b54c80a3f5264695047e4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:34.538609 17988 raft_consensus.cc:2258] T e8f12042b09947b6812f2d70ec1176e0 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:34.539189 17988 ts_tablet_manager.cc:1875] T e8f12042b09947b6812f2d70ec1176e0 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:34.540464 17987 ts_tablet_manager.cc:1888] T e8f12042b09947b6812f2d70ec1176e0 P 009f849ee7b54c80a3f5264695047e4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:34.540573 17987 log.cc:1196] T e8f12042b09947b6812f2d70ec1176e0 P 009f849ee7b54c80a3f5264695047e4d: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/wal/wals/e8f12042b09947b6812f2d70ec1176e0
I20240503 13:56:34.540818 17986 ts_tablet_manager.cc:1888] T e8f12042b09947b6812f2d70ec1176e0 P 76c386bef61a4bd5a907266a5118ffe2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:34.540913 17986 log.cc:1196] T e8f12042b09947b6812f2d70ec1176e0 P 76c386bef61a4bd5a907266a5118ffe2: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/wal/wals/e8f12042b09947b6812f2d70ec1176e0
I20240503 13:56:34.541134 17987 ts_tablet_manager.cc:1909] T e8f12042b09947b6812f2d70ec1176e0 P 009f849ee7b54c80a3f5264695047e4d: Deleting consensus metadata
I20240503 13:56:34.541182 17986 ts_tablet_manager.cc:1909] T e8f12042b09947b6812f2d70ec1176e0 P 76c386bef61a4bd5a907266a5118ffe2: Deleting consensus metadata
I20240503 13:56:34.541405 17988 ts_tablet_manager.cc:1888] T e8f12042b09947b6812f2d70ec1176e0 P 7690ee79577c45e3a2c8f8b552c0bff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:34.541478 17988 log.cc:1196] T e8f12042b09947b6812f2d70ec1176e0 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal/wals/e8f12042b09947b6812f2d70ec1176e0
I20240503 13:56:34.541769 17988 ts_tablet_manager.cc:1909] T e8f12042b09947b6812f2d70ec1176e0 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting consensus metadata
I20240503 13:56:34.541849 12896 catalog_manager.cc:4830] TS 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951): tablet e8f12042b09947b6812f2d70ec1176e0 (table rebalancer_dynamic_table_new_name [id=cf0f26e8e19d47e09ce29d5ec3ab1919]) successfully deleted
I20240503 13:56:34.541849 12899 catalog_manager.cc:4830] TS 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885): tablet e8f12042b09947b6812f2d70ec1176e0 (table rebalancer_dynamic_table_new_name [id=cf0f26e8e19d47e09ce29d5ec3ab1919]) successfully deleted
I20240503 13:56:34.542405 12896 catalog_manager.cc:4830] TS 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109): tablet e8f12042b09947b6812f2d70ec1176e0 (table rebalancer_dynamic_table_new_name [id=cf0f26e8e19d47e09ce29d5ec3ab1919]) successfully deleted
I20240503 13:56:34.611984 17385 raft_consensus.cc:482] T 3c197f47582546e39cda80c63bc98f23 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:34.612120 17385 raft_consensus.cc:504] T 3c197f47582546e39cda80c63bc98f23 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:34.612577 17385 leader_election.cc:288] T 3c197f47582546e39cda80c63bc98f23 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695), 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951)
I20240503 13:56:34.612857 13051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c197f47582546e39cda80c63bc98f23" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e" is_pre_election: true
I20240503 13:56:34.613003 13051 raft_consensus.cc:2454] T 3c197f47582546e39cda80c63bc98f23 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 0.
I20240503 13:56:34.613045 15624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c197f47582546e39cda80c63bc98f23" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d" is_pre_election: true
I20240503 13:56:34.613173 15624 raft_consensus.cc:2454] T 3c197f47582546e39cda80c63bc98f23 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 0.
I20240503 13:56:34.613476 15426 leader_election.cc:302] T 3c197f47582546e39cda80c63bc98f23 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 61b887f0d40d497badeca5604c1f0546; no voters:
I20240503 13:56:34.613703 17385 raft_consensus.cc:2781] T 3c197f47582546e39cda80c63bc98f23 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:34.613765 17385 raft_consensus.cc:482] T 3c197f47582546e39cda80c63bc98f23 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:34.613809 17385 raft_consensus.cc:3037] T 3c197f47582546e39cda80c63bc98f23 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:34.614689 17385 raft_consensus.cc:504] T 3c197f47582546e39cda80c63bc98f23 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:34.614974 17385 leader_election.cc:288] T 3c197f47582546e39cda80c63bc98f23 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 election: Requested vote from peers 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695), 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951)
I20240503 13:56:34.615291 15624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c197f47582546e39cda80c63bc98f23" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d"
I20240503 13:56:34.615412 15624 raft_consensus.cc:3037] T 3c197f47582546e39cda80c63bc98f23 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:34.615418 13051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c197f47582546e39cda80c63bc98f23" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e"
I20240503 13:56:34.615538 13051 raft_consensus.cc:3037] T 3c197f47582546e39cda80c63bc98f23 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:34.616518 13051 raft_consensus.cc:2454] T 3c197f47582546e39cda80c63bc98f23 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 1.
I20240503 13:56:34.616555 15624 raft_consensus.cc:2454] T 3c197f47582546e39cda80c63bc98f23 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 1.
I20240503 13:56:34.616834 15427 leader_election.cc:302] T 3c197f47582546e39cda80c63bc98f23 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 61b887f0d40d497badeca5604c1f0546; no voters:
I20240503 13:56:34.617002 17385 raft_consensus.cc:2781] T 3c197f47582546e39cda80c63bc98f23 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:34.617100 17385 raft_consensus.cc:686] T 3c197f47582546e39cda80c63bc98f23 P 61b887f0d40d497badeca5604c1f0546 [term 1 LEADER]: Becoming Leader. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Running, Role: LEADER
I20240503 13:56:34.617241 17385 consensus_queue.cc:237] T 3c197f47582546e39cda80c63bc98f23 P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:34.618928 12909 catalog_manager.cc:5484] T 3c197f47582546e39cda80c63bc98f23 P 61b887f0d40d497badeca5604c1f0546 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61b887f0d40d497badeca5604c1f0546 (127.6.98.69). New cstate: current_term: 1 leader_uuid: "61b887f0d40d497badeca5604c1f0546" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:34.624890 13185 raft_consensus.cc:1261] T 9eb0d911701b4b2bb33a6b7ab59f08aa P 8ccf5a4ac450414c9d3579bd5800b770 [term 2 FOLLOWER]: Refusing update from remote peer 61b887f0d40d497badeca5604c1f0546: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 13:56:34.625349 17385 consensus_queue.cc:1035] T 9eb0d911701b4b2bb33a6b7ab59f08aa P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:34.638353 12903 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:56:34.644097 15605 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:56:34.644253 13034 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:34.646447 15867 tablet_service.cc:1467] Processing CreateTablet for tablet a0bb459e3cfe4276833d30ca4e31bc90 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=36becc37cb0b4f4d93831bdbef7f85e5]), partition=
I20240503 13:56:34.646667 15736 tablet_service.cc:1467] Processing CreateTablet for tablet a0bb459e3cfe4276833d30ca4e31bc90 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=36becc37cb0b4f4d93831bdbef7f85e5]), partition=
I20240503 13:56:34.646732 15867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0bb459e3cfe4276833d30ca4e31bc90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:34.646875 15736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0bb459e3cfe4276833d30ca4e31bc90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:34.648958 17904 tablet_bootstrap.cc:492] T a0bb459e3cfe4276833d30ca4e31bc90 P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:34.649746 15605 tablet_service.cc:1467] Processing CreateTablet for tablet a0bb459e3cfe4276833d30ca4e31bc90 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=36becc37cb0b4f4d93831bdbef7f85e5]), partition=
I20240503 13:56:34.649845 17904 tablet_bootstrap.cc:654] T a0bb459e3cfe4276833d30ca4e31bc90 P e47105a0f46e4511b7112e571688a4c7: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:34.649928 15605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0bb459e3cfe4276833d30ca4e31bc90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:34.651651 17661 tablet_bootstrap.cc:492] T a0bb459e3cfe4276833d30ca4e31bc90 P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:34.652244 17904 tablet_bootstrap.cc:492] T a0bb459e3cfe4276833d30ca4e31bc90 P e47105a0f46e4511b7112e571688a4c7: No bootstrap required, opened a new log
I20240503 13:56:34.652314 17904 ts_tablet_manager.cc:1367] T a0bb459e3cfe4276833d30ca4e31bc90 P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:56:34.652761 17661 tablet_bootstrap.cc:654] T a0bb459e3cfe4276833d30ca4e31bc90 P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:34.652756 17904 raft_consensus.cc:348] T a0bb459e3cfe4276833d30ca4e31bc90 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:34.652863 17904 raft_consensus.cc:374] T a0bb459e3cfe4276833d30ca4e31bc90 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:34.652895 17904 raft_consensus.cc:729] T a0bb459e3cfe4276833d30ca4e31bc90 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:34.653014 17904 consensus_queue.cc:260] T a0bb459e3cfe4276833d30ca4e31bc90 P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:34.653864 15472 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:56:34.656414 17567 tablet_bootstrap.cc:492] T a0bb459e3cfe4276833d30ca4e31bc90 P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:34.657238 17567 tablet_bootstrap.cc:654] T a0bb459e3cfe4276833d30ca4e31bc90 P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:34.659045 17904 ts_tablet_manager.cc:1398] T a0bb459e3cfe4276833d30ca4e31bc90 P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240503 13:56:34.661648 15867 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:56:34.665665 17661 tablet_bootstrap.cc:492] T a0bb459e3cfe4276833d30ca4e31bc90 P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:34.665772 17661 ts_tablet_manager.cc:1367] T a0bb459e3cfe4276833d30ca4e31bc90 P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.001s
I20240503 13:56:34.666936 17661 raft_consensus.cc:348] T a0bb459e3cfe4276833d30ca4e31bc90 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:34.667073 17661 raft_consensus.cc:374] T a0bb459e3cfe4276833d30ca4e31bc90 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:34.667106 17661 raft_consensus.cc:729] T a0bb459e3cfe4276833d30ca4e31bc90 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:34.667234 17661 consensus_queue.cc:260] T a0bb459e3cfe4276833d30ca4e31bc90 P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:34.667511 17661 ts_tablet_manager.cc:1398] T a0bb459e3cfe4276833d30ca4e31bc90 P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:56:34.668833 13424 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:56:34.672897 17567 tablet_bootstrap.cc:492] T a0bb459e3cfe4276833d30ca4e31bc90 P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:34.672991 17567 ts_tablet_manager.cc:1367] T a0bb459e3cfe4276833d30ca4e31bc90 P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240503 13:56:34.673427 17567 raft_consensus.cc:348] T a0bb459e3cfe4276833d30ca4e31bc90 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:34.673534 17567 raft_consensus.cc:374] T a0bb459e3cfe4276833d30ca4e31bc90 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:34.673560 17567 raft_consensus.cc:729] T a0bb459e3cfe4276833d30ca4e31bc90 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:34.673668 17567 consensus_queue.cc:260] T a0bb459e3cfe4276833d30ca4e31bc90 P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:34.673852 15736 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:56:34.673907 17567 ts_tablet_manager.cc:1398] T a0bb459e3cfe4276833d30ca4e31bc90 P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:34.674180 13165 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:34.691848 13295 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:56:34.752928 13314 raft_consensus.cc:1261] T 9eb0d911701b4b2bb33a6b7ab59f08aa P ba0194098aa84449ae17e8a0d4c43703 [term 2 FOLLOWER]: Refusing update from remote peer 61b887f0d40d497badeca5604c1f0546: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 13:56:34.753410 17385 consensus_queue.cc:1035] T 9eb0d911701b4b2bb33a6b7ab59f08aa P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:34.755268 13316 consensus_queue.cc:237] T 4939335096b94444a61503841e2e81eb P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:34.756758 13445 raft_consensus.cc:1261] T 9eb0d911701b4b2bb33a6b7ab59f08aa P 7690ee79577c45e3a2c8f8b552c0bff8 [term 2 FOLLOWER]: Refusing update from remote peer 61b887f0d40d497badeca5604c1f0546: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 13:56:34.757020 13185 raft_consensus.cc:1261] T 4939335096b94444a61503841e2e81eb P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:56:34.757160 13051 raft_consensus.cc:1261] T 4939335096b94444a61503841e2e81eb P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:56:34.757475 17858 consensus_queue.cc:1035] T 9eb0d911701b4b2bb33a6b7ab59f08aa P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:34.757683 17555 consensus_queue.cc:1035] T 4939335096b94444a61503841e2e81eb P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 13:56:34.757921 13249 consensus_peers.cc:485] T 4939335096b94444a61503841e2e81eb P ba0194098aa84449ae17e8a0d4c43703 -> Peer 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951): Couldn't send request to peer 009f849ee7b54c80a3f5264695047e4d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4939335096b94444a61503841e2e81eb. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:34.758020 17555 consensus_queue.cc:1035] T 4939335096b94444a61503841e2e81eb P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:34.762650 16890 raft_consensus.cc:2932] T 4939335096b94444a61503841e2e81eb P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 009f849ee7b54c80a3f5264695047e4d (127.6.98.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:34.764605 13051 raft_consensus.cc:2932] T 4939335096b94444a61503841e2e81eb P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 009f849ee7b54c80a3f5264695047e4d (127.6.98.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:34.764636 13185 raft_consensus.cc:2932] T 4939335096b94444a61503841e2e81eb P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 009f849ee7b54c80a3f5264695047e4d (127.6.98.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:34.765379 12903 catalog_manager.cc:5484] T 4939335096b94444a61503841e2e81eb P ba0194098aa84449ae17e8a0d4c43703 reported cstate change: config changed from index -1 to 4, VOTER 009f849ee7b54c80a3f5264695047e4d (127.6.98.70) added. New cstate: current_term: 1 leader_uuid: "ba0194098aa84449ae17e8a0d4c43703" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:34.787413 13185 consensus_queue.cc:237] T a3c4cbf779554d8b84a48de78a9c392e P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:34.789009 15493 raft_consensus.cc:1261] T a3c4cbf779554d8b84a48de78a9c392e P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Refusing update from remote peer 8ccf5a4ac450414c9d3579bd5800b770: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 13:56:34.789292 13119 consensus_peers.cc:485] T a3c4cbf779554d8b84a48de78a9c392e P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695): Couldn't send request to peer 60ba3ffe3b7746469834cd91c84be14e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a3c4cbf779554d8b84a48de78a9c392e. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:34.789449 18048 consensus_queue.cc:1035] T a3c4cbf779554d8b84a48de78a9c392e P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:56:34.796794 15889 raft_consensus.cc:1261] T a3c4cbf779554d8b84a48de78a9c392e P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Refusing update from remote peer 8ccf5a4ac450414c9d3579bd5800b770: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:34.797312 18048 consensus_queue.cc:1035] T a3c4cbf779554d8b84a48de78a9c392e P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:56:34.799270 18005 raft_consensus.cc:2932] T a3c4cbf779554d8b84a48de78a9c392e P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } }
I20240503 13:56:34.799662 15889 raft_consensus.cc:2932] T a3c4cbf779554d8b84a48de78a9c392e P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } }
I20240503 13:56:34.801211 15493 raft_consensus.cc:2932] T a3c4cbf779554d8b84a48de78a9c392e P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } }
I20240503 13:56:34.802477 12904 catalog_manager.cc:5484] T a3c4cbf779554d8b84a48de78a9c392e P e47105a0f46e4511b7112e571688a4c7 reported cstate change: config changed from index -1 to 2, VOTER 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65) added. New cstate: current_term: 1 leader_uuid: "8ccf5a4ac450414c9d3579bd5800b770" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } }
I20240503 13:56:34.825189 15889 consensus_queue.cc:237] T 0f9224c2e3b946d993a5fa5c95a1d681 P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:34.826984 15624 raft_consensus.cc:1261] T 0f9224c2e3b946d993a5fa5c95a1d681 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Refusing update from remote peer e47105a0f46e4511b7112e571688a4c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:34.827018 15757 raft_consensus.cc:1261] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Refusing update from remote peer e47105a0f46e4511b7112e571688a4c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 13:56:34.827096 15822 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P e47105a0f46e4511b7112e571688a4c7 -> Peer 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109): Couldn't send request to peer 7690ee79577c45e3a2c8f8b552c0bff8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f9224c2e3b946d993a5fa5c95a1d681. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:34.827411 16404 consensus_queue.cc:1035] T 0f9224c2e3b946d993a5fa5c95a1d681 P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:56:34.827608 17953 consensus_queue.cc:1035] T 0f9224c2e3b946d993a5fa5c95a1d681 P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:56:34.829432 16404 raft_consensus.cc:2932] T 0f9224c2e3b946d993a5fa5c95a1d681 P e47105a0f46e4511b7112e571688a4c7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } }
I20240503 13:56:34.829883 15624 raft_consensus.cc:2932] T 0f9224c2e3b946d993a5fa5c95a1d681 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } }
I20240503 13:56:34.832265 15757 raft_consensus.cc:2932] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } }
I20240503 13:56:34.832559 12910 catalog_manager.cc:5484] T 0f9224c2e3b946d993a5fa5c95a1d681 P e47105a0f46e4511b7112e571688a4c7 reported cstate change: config changed from index -1 to 2, VOTER 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68) added. New cstate: current_term: 1 leader_uuid: "e47105a0f46e4511b7112e571688a4c7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } }
I20240503 13:56:34.847126 16404 consensus_queue.cc:1035] T f58fc3bc579748bb88de34612808ccda P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:34.850605 18071 consensus_queue.cc:1035] T f58fc3bc579748bb88de34612808ccda P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:34.854279 13052 consensus_queue.cc:237] T 66121c347e6c443dac7c5123826a7ae1 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:34.855620 13316 raft_consensus.cc:1261] T 66121c347e6c443dac7c5123826a7ae1 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Refusing update from remote peer 60ba3ffe3b7746469834cd91c84be14e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:56:34.855713 13185 raft_consensus.cc:1261] T 66121c347e6c443dac7c5123826a7ae1 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Refusing update from remote peer 60ba3ffe3b7746469834cd91c84be14e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 13:56:34.855894 12990 consensus_peers.cc:485] T 66121c347e6c443dac7c5123826a7ae1 P 60ba3ffe3b7746469834cd91c84be14e -> Peer 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153): Couldn't send request to peer 61b887f0d40d497badeca5604c1f0546. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 66121c347e6c443dac7c5123826a7ae1. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:34.856127 18091 consensus_queue.cc:1035] T 66121c347e6c443dac7c5123826a7ae1 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:34.856376 17343 consensus_queue.cc:1035] T 66121c347e6c443dac7c5123826a7ae1 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:34.858142 17343 raft_consensus.cc:2932] T 66121c347e6c443dac7c5123826a7ae1 P 60ba3ffe3b7746469834cd91c84be14e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 61b887f0d40d497badeca5604c1f0546 (127.6.98.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:34.858583 13316 raft_consensus.cc:2932] T 66121c347e6c443dac7c5123826a7ae1 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 61b887f0d40d497badeca5604c1f0546 (127.6.98.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:34.860685 12909 catalog_manager.cc:5484] T 66121c347e6c443dac7c5123826a7ae1 P 60ba3ffe3b7746469834cd91c84be14e reported cstate change: config changed from index -1 to 4, VOTER 61b887f0d40d497badeca5604c1f0546 (127.6.98.69) added. New cstate: current_term: 1 leader_uuid: "60ba3ffe3b7746469834cd91c84be14e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:34.860854 13185 raft_consensus.cc:2932] T 66121c347e6c443dac7c5123826a7ae1 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 61b887f0d40d497badeca5604c1f0546 (127.6.98.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:34.880048 17953 raft_consensus.cc:482] T a0bb459e3cfe4276833d30ca4e31bc90 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:34.880172 17953 raft_consensus.cc:504] T a0bb459e3cfe4276833d30ca4e31bc90 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:34.880640 17953 leader_election.cc:288] T a0bb459e3cfe4276833d30ca4e31bc90 P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951), 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885)
I20240503 13:56:34.880995 15624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0bb459e3cfe4276833d30ca4e31bc90" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d" is_pre_election: true
I20240503 13:56:34.881044 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0bb459e3cfe4276833d30ca4e31bc90" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2" is_pre_election: true
I20240503 13:56:34.881139 15624 raft_consensus.cc:2454] T a0bb459e3cfe4276833d30ca4e31bc90 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 0.
I20240503 13:56:34.881191 15757 raft_consensus.cc:2454] T a0bb459e3cfe4276833d30ca4e31bc90 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 0.
I20240503 13:56:34.881390 15822 leader_election.cc:302] T a0bb459e3cfe4276833d30ca4e31bc90 P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:34.881597 17953 raft_consensus.cc:2781] T a0bb459e3cfe4276833d30ca4e31bc90 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:34.881645 17953 raft_consensus.cc:482] T a0bb459e3cfe4276833d30ca4e31bc90 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:34.881681 17953 raft_consensus.cc:3037] T a0bb459e3cfe4276833d30ca4e31bc90 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:34.882254 15493 consensus_queue.cc:237] T fdc5f76f63ec4528ad57202555caf243 P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:34.882484 17953 raft_consensus.cc:504] T a0bb459e3cfe4276833d30ca4e31bc90 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:34.882804 17953 leader_election.cc:288] T a0bb459e3cfe4276833d30ca4e31bc90 P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 election: Requested vote from peers 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951), 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885)
I20240503 13:56:34.883131 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0bb459e3cfe4276833d30ca4e31bc90" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2"
I20240503 13:56:34.883278 15757 raft_consensus.cc:3037] T a0bb459e3cfe4276833d30ca4e31bc90 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:34.883273 15624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0bb459e3cfe4276833d30ca4e31bc90" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d"
I20240503 13:56:34.883386 15624 raft_consensus.cc:3037] T a0bb459e3cfe4276833d30ca4e31bc90 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
W20240503 13:56:34.884008 15429 consensus_peers.cc:485] T fdc5f76f63ec4528ad57202555caf243 P 61b887f0d40d497badeca5604c1f0546 -> Peer 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387): Couldn't send request to peer 8ccf5a4ac450414c9d3579bd5800b770. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fdc5f76f63ec4528ad57202555caf243. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:34.884114 15757 raft_consensus.cc:2454] T a0bb459e3cfe4276833d30ca4e31bc90 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 1.
I20240503 13:56:34.884282 15624 raft_consensus.cc:2454] T a0bb459e3cfe4276833d30ca4e31bc90 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 1.
I20240503 13:56:34.884361 15825 leader_election.cc:302] T a0bb459e3cfe4276833d30ca4e31bc90 P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76c386bef61a4bd5a907266a5118ffe2, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:34.884657 17953 raft_consensus.cc:2781] T a0bb459e3cfe4276833d30ca4e31bc90 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:34.884742 15757 raft_consensus.cc:1261] T fdc5f76f63ec4528ad57202555caf243 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Refusing update from remote peer 61b887f0d40d497badeca5604c1f0546: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:34.884768 17953 raft_consensus.cc:686] T a0bb459e3cfe4276833d30ca4e31bc90 P e47105a0f46e4511b7112e571688a4c7 [term 1 LEADER]: Becoming Leader. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Running, Role: LEADER
I20240503 13:56:34.884917 17953 consensus_queue.cc:237] T a0bb459e3cfe4276833d30ca4e31bc90 P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:34.885149 18023 consensus_queue.cc:1035] T fdc5f76f63ec4528ad57202555caf243 P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:56:34.886070 13052 raft_consensus.cc:1261] T fdc5f76f63ec4528ad57202555caf243 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Refusing update from remote peer 61b887f0d40d497badeca5604c1f0546: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:34.886467 18023 consensus_queue.cc:1035] T fdc5f76f63ec4528ad57202555caf243 P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:56:34.886906 12903 catalog_manager.cc:5484] T a0bb459e3cfe4276833d30ca4e31bc90 P e47105a0f46e4511b7112e571688a4c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e47105a0f46e4511b7112e571688a4c7 (127.6.98.72). New cstate: current_term: 1 leader_uuid: "e47105a0f46e4511b7112e571688a4c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:34.888176 17858 raft_consensus.cc:2932] T fdc5f76f63ec4528ad57202555caf243 P 61b887f0d40d497badeca5604c1f0546 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } }
I20240503 13:56:34.889153 13052 raft_consensus.cc:2932] T fdc5f76f63ec4528ad57202555caf243 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } }
I20240503 13:56:34.890295 15757 raft_consensus.cc:2932] T fdc5f76f63ec4528ad57202555caf243 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } }
I20240503 13:56:34.891892 12904 catalog_manager.cc:5484] T fdc5f76f63ec4528ad57202555caf243 P 60ba3ffe3b7746469834cd91c84be14e reported cstate change: config changed from index -1 to 2, VOTER 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66) added. New cstate: current_term: 1 leader_uuid: "61b887f0d40d497badeca5604c1f0546" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } }
I20240503 13:56:34.912644 13052 consensus_queue.cc:237] T 855c5fc4cefb48b58533f26d62595465 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:34.914698 13445 raft_consensus.cc:1261] T 855c5fc4cefb48b58533f26d62595465 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer 60ba3ffe3b7746469834cd91c84be14e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:34.914880 15889 raft_consensus.cc:1261] T 855c5fc4cefb48b58533f26d62595465 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Refusing update from remote peer 60ba3ffe3b7746469834cd91c84be14e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:34.915349 18091 consensus_queue.cc:1035] T 855c5fc4cefb48b58533f26d62595465 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:56:34.915674 17343 consensus_queue.cc:1035] T 855c5fc4cefb48b58533f26d62595465 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 13:56:34.918548 12987 consensus_peers.cc:485] T 855c5fc4cefb48b58533f26d62595465 P 60ba3ffe3b7746469834cd91c84be14e -> Peer 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951): Couldn't send request to peer 009f849ee7b54c80a3f5264695047e4d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 855c5fc4cefb48b58533f26d62595465. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:34.918642 18091 raft_consensus.cc:2932] T 855c5fc4cefb48b58533f26d62595465 P 60ba3ffe3b7746469834cd91c84be14e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 009f849ee7b54c80a3f5264695047e4d (127.6.98.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:34.919085 13445 raft_consensus.cc:2932] T 855c5fc4cefb48b58533f26d62595465 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 009f849ee7b54c80a3f5264695047e4d (127.6.98.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:34.920719 15889 raft_consensus.cc:2932] T 855c5fc4cefb48b58533f26d62595465 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 009f849ee7b54c80a3f5264695047e4d (127.6.98.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:34.921689 12909 catalog_manager.cc:5484] T 855c5fc4cefb48b58533f26d62595465 P 60ba3ffe3b7746469834cd91c84be14e reported cstate change: config changed from index -1 to 2, VOTER 009f849ee7b54c80a3f5264695047e4d (127.6.98.70) added. New cstate: current_term: 1 leader_uuid: "60ba3ffe3b7746469834cd91c84be14e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:35.000384 12910 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:56:35.006071 15472 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:56:35.012022 15624 raft_consensus.cc:1261] T a0bb459e3cfe4276833d30ca4e31bc90 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Refusing update from remote peer e47105a0f46e4511b7112e571688a4c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:35.012022 15757 raft_consensus.cc:1261] T a0bb459e3cfe4276833d30ca4e31bc90 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Refusing update from remote peer e47105a0f46e4511b7112e571688a4c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:35.012615 17953 consensus_queue.cc:1035] T a0bb459e3cfe4276833d30ca4e31bc90 P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:35.012977 15605 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:56:35.013080 17953 consensus_queue.cc:1035] T a0bb459e3cfe4276833d30ca4e31bc90 P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:35.016929 13424 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:56:35.018580 18193 tablet.cc:1722] T a0bb459e3cfe4276833d30ca4e31bc90 P 76c386bef61a4bd5a907266a5118ffe2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:35.018972 18193 tablet.cc:1620] T a0bb459e3cfe4276833d30ca4e31bc90 P 76c386bef61a4bd5a907266a5118ffe2: MemRowSet was empty: no flush needed.
I20240503 13:56:35.019112 18192 tablet.cc:1722] T a0bb459e3cfe4276833d30ca4e31bc90 P e47105a0f46e4511b7112e571688a4c7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:35.019563 18192 tablet.cc:1620] T a0bb459e3cfe4276833d30ca4e31bc90 P e47105a0f46e4511b7112e571688a4c7: MemRowSet was empty: no flush needed.
I20240503 13:56:35.020469 18071 consensus_queue.cc:1035] T 4b4b26adad184e1ea57c00a44635679c P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:35.022159 12909 catalog_manager.cc:5576] TS 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885) does not have the latest schema for tablet a0bb459e3cfe4276833d30ca4e31bc90 (table rebalancer_dynamic_table [id=36becc37cb0b4f4d93831bdbef7f85e5]). Expected version 1 got 0
I20240503 13:56:35.027544 15736 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:56:35.032200 13165 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:35.033135 12909 catalog_manager.cc:5576] TS 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951) does not have the latest schema for tablet a0bb459e3cfe4276833d30ca4e31bc90 (table rebalancer_dynamic_table [id=36becc37cb0b4f4d93831bdbef7f85e5]). Expected version 1 got 0
I20240503 13:56:35.038864 13034 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:35.041816 13295 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:56:35.042025 16404 consensus_queue.cc:1035] T 4b4b26adad184e1ea57c00a44635679c P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:35.042567 15867 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:35.052986 12910 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=36becc37cb0b4f4d93831bdbef7f85e5] alter complete (version 1)
I20240503 13:56:35.054687 12910 catalog_manager.cc:5576] TS 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951) does not have the latest schema for tablet a0bb459e3cfe4276833d30ca4e31bc90 (table rebalancer_dynamic_table [id=36becc37cb0b4f4d93831bdbef7f85e5]). Expected version 1 got 0
I20240503 13:56:35.058089 18195 tablet.cc:1722] T a0bb459e3cfe4276833d30ca4e31bc90 P 009f849ee7b54c80a3f5264695047e4d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:35.058584 18195 tablet.cc:1620] T a0bb459e3cfe4276833d30ca4e31bc90 P 009f849ee7b54c80a3f5264695047e4d: MemRowSet was empty: no flush needed.
I20240503 13:56:35.137249 18023 consensus_queue.cc:1035] T 3c197f47582546e39cda80c63bc98f23 P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:35.148669 17858 consensus_queue.cc:1035] T 3c197f47582546e39cda80c63bc98f23 P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:35.155041 12906 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:56:35.162214 18192 tablet.cc:1722] T a0bb459e3cfe4276833d30ca4e31bc90 P e47105a0f46e4511b7112e571688a4c7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:35.162580 18192 tablet.cc:1620] T a0bb459e3cfe4276833d30ca4e31bc90 P e47105a0f46e4511b7112e571688a4c7: MemRowSet was empty: no flush needed.
I20240503 13:56:35.163724 18195 tablet.cc:1722] T a0bb459e3cfe4276833d30ca4e31bc90 P 009f849ee7b54c80a3f5264695047e4d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:35.164047 18195 tablet.cc:1620] T a0bb459e3cfe4276833d30ca4e31bc90 P 009f849ee7b54c80a3f5264695047e4d: MemRowSet was empty: no flush needed.
I20240503 13:56:35.164206 18193 tablet.cc:1722] T a0bb459e3cfe4276833d30ca4e31bc90 P 76c386bef61a4bd5a907266a5118ffe2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:35.164474 18193 tablet.cc:1620] T a0bb459e3cfe4276833d30ca4e31bc90 P 76c386bef61a4bd5a907266a5118ffe2: MemRowSet was empty: no flush needed.
I20240503 13:56:35.169220 12910 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=36becc37cb0b4f4d93831bdbef7f85e5] alter complete (version 2)
I20240503 13:56:35.178321 13034 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:35.179337 15472 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:56:35.184355 13424 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:56:35.188043 15605 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:56:35.193480 15736 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:56:35.197067 13165 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:35.203426 12905 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_extra_table_16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:56:35.209836 15867 tablet_service.cc:1467] Processing CreateTablet for tablet 557f14aaf5564ebaa461720a4d7997ff (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=2787ed323f264ddfbb66f82ba4e58775]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:35.210151 15867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 557f14aaf5564ebaa461720a4d7997ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:35.210120 13295 tablet_service.cc:1467] Processing CreateTablet for tablet 557f14aaf5564ebaa461720a4d7997ff (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=2787ed323f264ddfbb66f82ba4e58775]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:35.210352 13295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 557f14aaf5564ebaa461720a4d7997ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:35.210500 15605 tablet_service.cc:1467] Processing CreateTablet for tablet 557f14aaf5564ebaa461720a4d7997ff (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=2787ed323f264ddfbb66f82ba4e58775]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:35.210770 15605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 557f14aaf5564ebaa461720a4d7997ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:35.213095 13165 tablet_service.cc:1467] Processing CreateTablet for tablet d3849e4effe64656bd8b0f504c9dc17a (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=2787ed323f264ddfbb66f82ba4e58775]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:35.213342 13165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3849e4effe64656bd8b0f504c9dc17a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:35.215597 13164 tablet_service.cc:1467] Processing CreateTablet for tablet 63290f22e28b4329901dfc7ea6da0cc7 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=2787ed323f264ddfbb66f82ba4e58775]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:35.215773 18253 tablet_bootstrap.cc:492] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:35.215801 13164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63290f22e28b4329901dfc7ea6da0cc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:35.216789 18253 tablet_bootstrap.cc:654] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:35.221369 15736 tablet_service.cc:1467] Processing CreateTablet for tablet 63290f22e28b4329901dfc7ea6da0cc7 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=2787ed323f264ddfbb66f82ba4e58775]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:35.221676 15736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63290f22e28b4329901dfc7ea6da0cc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:35.222479 15604 tablet_service.cc:1467] Processing CreateTablet for tablet 63290f22e28b4329901dfc7ea6da0cc7 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=2787ed323f264ddfbb66f82ba4e58775]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:35.222566 15867 tablet_service.cc:1467] Processing CreateTablet for tablet d3849e4effe64656bd8b0f504c9dc17a (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=2787ed323f264ddfbb66f82ba4e58775]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:35.222721 15604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63290f22e28b4329901dfc7ea6da0cc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:35.222803 15867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3849e4effe64656bd8b0f504c9dc17a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:35.224396 18251 tablet_bootstrap.cc:492] T 557f14aaf5564ebaa461720a4d7997ff P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:35.226652 15605 tablet_service.cc:1467] Processing CreateTablet for tablet d3849e4effe64656bd8b0f504c9dc17a (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=2787ed323f264ddfbb66f82ba4e58775]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:35.226774 13295 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:56:35.226861 15605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3849e4effe64656bd8b0f504c9dc17a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:35.226401 15867 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:35.225382 18251 tablet_bootstrap.cc:654] T 557f14aaf5564ebaa461720a4d7997ff P e47105a0f46e4511b7112e571688a4c7: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:35.237622 18253 tablet_bootstrap.cc:492] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:35.237737 18253 ts_tablet_manager.cc:1367] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20240503 13:56:35.238190 18253 raft_consensus.cc:348] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:35.238294 18253 raft_consensus.cc:374] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:35.238319 18253 raft_consensus.cc:729] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:35.238430 18253 consensus_queue.cc:260] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:35.238692 18253 ts_tablet_manager.cc:1398] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:35.238852 18253 tablet_bootstrap.cc:492] T 63290f22e28b4329901dfc7ea6da0cc7 P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:35.239820 18253 tablet_bootstrap.cc:654] T 63290f22e28b4329901dfc7ea6da0cc7 P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:35.241624 18256 tablet_bootstrap.cc:492] T 63290f22e28b4329901dfc7ea6da0cc7 P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:35.242913 18256 tablet_bootstrap.cc:654] T 63290f22e28b4329901dfc7ea6da0cc7 P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:35.243427 18253 tablet_bootstrap.cc:492] T 63290f22e28b4329901dfc7ea6da0cc7 P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:35.243520 18253 ts_tablet_manager.cc:1367] T 63290f22e28b4329901dfc7ea6da0cc7 P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:56:35.243942 18253 raft_consensus.cc:348] T 63290f22e28b4329901dfc7ea6da0cc7 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:35.244045 18253 raft_consensus.cc:374] T 63290f22e28b4329901dfc7ea6da0cc7 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:35.244087 18253 raft_consensus.cc:729] T 63290f22e28b4329901dfc7ea6da0cc7 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:35.244192 18253 consensus_queue.cc:260] T 63290f22e28b4329901dfc7ea6da0cc7 P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:35.244422 18253 ts_tablet_manager.cc:1398] T 63290f22e28b4329901dfc7ea6da0cc7 P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:35.244622 18253 tablet_bootstrap.cc:492] T d3849e4effe64656bd8b0f504c9dc17a P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:35.246851 18253 tablet_bootstrap.cc:654] T d3849e4effe64656bd8b0f504c9dc17a P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:35.249862 18251 tablet_bootstrap.cc:492] T 557f14aaf5564ebaa461720a4d7997ff P e47105a0f46e4511b7112e571688a4c7: No bootstrap required, opened a new log
I20240503 13:56:35.250011 18251 ts_tablet_manager.cc:1367] T 557f14aaf5564ebaa461720a4d7997ff P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.037s user 0.002s sys 0.000s
I20240503 13:56:35.250623 18251 raft_consensus.cc:348] T 557f14aaf5564ebaa461720a4d7997ff P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:35.251014 18251 raft_consensus.cc:374] T 557f14aaf5564ebaa461720a4d7997ff P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:35.251098 18251 raft_consensus.cc:729] T 557f14aaf5564ebaa461720a4d7997ff P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:35.251281 18251 consensus_queue.cc:260] T 557f14aaf5564ebaa461720a4d7997ff P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:35.252163 18251 ts_tablet_manager.cc:1398] T 557f14aaf5564ebaa461720a4d7997ff P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:35.252386 18251 tablet_bootstrap.cc:492] T d3849e4effe64656bd8b0f504c9dc17a P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:35.253609 18251 tablet_bootstrap.cc:654] T d3849e4effe64656bd8b0f504c9dc17a P e47105a0f46e4511b7112e571688a4c7: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:35.254276 18253 tablet_bootstrap.cc:492] T d3849e4effe64656bd8b0f504c9dc17a P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:35.254361 18253 ts_tablet_manager.cc:1367] T d3849e4effe64656bd8b0f504c9dc17a P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 13:56:35.254830 18253 raft_consensus.cc:348] T d3849e4effe64656bd8b0f504c9dc17a P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:35.254930 18253 raft_consensus.cc:374] T d3849e4effe64656bd8b0f504c9dc17a P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:35.254961 18253 raft_consensus.cc:729] T d3849e4effe64656bd8b0f504c9dc17a P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:35.255075 18253 consensus_queue.cc:260] T d3849e4effe64656bd8b0f504c9dc17a P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:35.255327 18253 ts_tablet_manager.cc:1398] T d3849e4effe64656bd8b0f504c9dc17a P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:35.263201 18252 tablet_bootstrap.cc:492] T 557f14aaf5564ebaa461720a4d7997ff P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:35.264710 18252 tablet_bootstrap.cc:654] T 557f14aaf5564ebaa461720a4d7997ff P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:35.265568 18254 tablet_bootstrap.cc:492] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:35.265841 18252 tablet_bootstrap.cc:492] T 557f14aaf5564ebaa461720a4d7997ff P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:35.265931 18252 ts_tablet_manager.cc:1367] T 557f14aaf5564ebaa461720a4d7997ff P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:56:35.265959 18251 tablet_bootstrap.cc:492] T d3849e4effe64656bd8b0f504c9dc17a P e47105a0f46e4511b7112e571688a4c7: No bootstrap required, opened a new log
I20240503 13:56:35.266072 18251 ts_tablet_manager.cc:1367] T d3849e4effe64656bd8b0f504c9dc17a P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240503 13:56:35.266316 18252 raft_consensus.cc:348] T 557f14aaf5564ebaa461720a4d7997ff P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:35.266412 18252 raft_consensus.cc:374] T 557f14aaf5564ebaa461720a4d7997ff P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:35.266440 18252 raft_consensus.cc:729] T 557f14aaf5564ebaa461720a4d7997ff P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:35.266482 18254 tablet_bootstrap.cc:654] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:35.266517 18251 raft_consensus.cc:348] T d3849e4effe64656bd8b0f504c9dc17a P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:35.266635 18251 raft_consensus.cc:374] T d3849e4effe64656bd8b0f504c9dc17a P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:35.266623 18252 consensus_queue.cc:260] T 557f14aaf5564ebaa461720a4d7997ff P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:35.266664 18251 raft_consensus.cc:729] T d3849e4effe64656bd8b0f504c9dc17a P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:35.266778 18251 consensus_queue.cc:260] T d3849e4effe64656bd8b0f504c9dc17a P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:35.266882 18252 ts_tablet_manager.cc:1398] T 557f14aaf5564ebaa461720a4d7997ff P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:35.267033 18251 ts_tablet_manager.cc:1398] T d3849e4effe64656bd8b0f504c9dc17a P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:35.269582 18254 tablet_bootstrap.cc:492] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:35.273438 18254 ts_tablet_manager.cc:1367] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 13:56:35.273437 12905 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:56:35.273963 18254 raft_consensus.cc:348] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:35.274118 18254 raft_consensus.cc:374] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:35.274184 18254 raft_consensus.cc:729] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:35.274341 18254 consensus_queue.cc:260] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:35.274619 18254 ts_tablet_manager.cc:1398] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:35.275898 18256 tablet_bootstrap.cc:492] T 63290f22e28b4329901dfc7ea6da0cc7 P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:35.276007 18256 ts_tablet_manager.cc:1367] T 63290f22e28b4329901dfc7ea6da0cc7 P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.034s user 0.002s sys 0.000s
I20240503 13:56:35.276451 18256 raft_consensus.cc:348] T 63290f22e28b4329901dfc7ea6da0cc7 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:35.276572 18256 raft_consensus.cc:374] T 63290f22e28b4329901dfc7ea6da0cc7 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:35.276605 18256 raft_consensus.cc:729] T 63290f22e28b4329901dfc7ea6da0cc7 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:35.276674 18254 tablet_bootstrap.cc:492] T 63290f22e28b4329901dfc7ea6da0cc7 P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:35.276726 18256 consensus_queue.cc:260] T 63290f22e28b4329901dfc7ea6da0cc7 P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:35.277001 18256 ts_tablet_manager.cc:1398] T 63290f22e28b4329901dfc7ea6da0cc7 P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:35.277624 18254 tablet_bootstrap.cc:654] T 63290f22e28b4329901dfc7ea6da0cc7 P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:35.278606 18254 tablet_bootstrap.cc:492] T 63290f22e28b4329901dfc7ea6da0cc7 P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:35.278709 18254 ts_tablet_manager.cc:1367] T 63290f22e28b4329901dfc7ea6da0cc7 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:56:35.279181 18254 raft_consensus.cc:348] T 63290f22e28b4329901dfc7ea6da0cc7 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:35.279304 18254 raft_consensus.cc:374] T 63290f22e28b4329901dfc7ea6da0cc7 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:35.279356 18254 raft_consensus.cc:729] T 63290f22e28b4329901dfc7ea6da0cc7 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:35.279481 18254 consensus_queue.cc:260] T 63290f22e28b4329901dfc7ea6da0cc7 P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:35.279484 18192 tablet.cc:1722] T a0bb459e3cfe4276833d30ca4e31bc90 P e47105a0f46e4511b7112e571688a4c7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:35.279803 18254 ts_tablet_manager.cc:1398] T 63290f22e28b4329901dfc7ea6da0cc7 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:35.280213 18193 tablet.cc:1722] T a0bb459e3cfe4276833d30ca4e31bc90 P 76c386bef61a4bd5a907266a5118ffe2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:35.282361 18195 tablet.cc:1722] T a0bb459e3cfe4276833d30ca4e31bc90 P 009f849ee7b54c80a3f5264695047e4d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:35.285476 12905 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table_new_name [id=36becc37cb0b4f4d93831bdbef7f85e5] alter complete (version 3)
I20240503 13:56:35.287140 18264 ts_tablet_manager.cc:902] T 66121c347e6c443dac7c5123826a7ae1 P 61b887f0d40d497badeca5604c1f0546: Initiating tablet copy from peer 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:35.287915 18264 tablet_copy_client.cc:308] T 66121c347e6c443dac7c5123826a7ae1 P 61b887f0d40d497badeca5604c1f0546: tablet copy: Beginning tablet copy session from remote peer at address 127.6.98.65:34695
I20240503 13:56:35.288499 13074 tablet_copy_service.cc:140] P 60ba3ffe3b7746469834cd91c84be14e: Received BeginTabletCopySession request for tablet 66121c347e6c443dac7c5123826a7ae1 from peer 61b887f0d40d497badeca5604c1f0546 ({username='slave'} at 127.6.98.69:58129)
I20240503 13:56:35.288597 13074 tablet_copy_service.cc:161] P 60ba3ffe3b7746469834cd91c84be14e: Beginning new tablet copy session on tablet 66121c347e6c443dac7c5123826a7ae1 from peer 61b887f0d40d497badeca5604c1f0546 at {username='slave'} at 127.6.98.69:58129: session id = 61b887f0d40d497badeca5604c1f0546-66121c347e6c443dac7c5123826a7ae1
I20240503 13:56:35.289534 13074 tablet_copy_source_session.cc:205] T 66121c347e6c443dac7c5123826a7ae1 P 60ba3ffe3b7746469834cd91c84be14e: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:56:35.291489 18264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66121c347e6c443dac7c5123826a7ae1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:35.293747 18264 tablet_copy_client.cc:783] T 66121c347e6c443dac7c5123826a7ae1 P 61b887f0d40d497badeca5604c1f0546: tablet copy: Starting download of 0 data blocks...
I20240503 13:56:35.293980 18264 tablet_copy_client.cc:647] T 66121c347e6c443dac7c5123826a7ae1 P 61b887f0d40d497badeca5604c1f0546: tablet copy: Starting download of 1 WAL segments...
I20240503 13:56:35.295295 18264 tablet_copy_client.cc:523] T 66121c347e6c443dac7c5123826a7ae1 P 61b887f0d40d497badeca5604c1f0546: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:56:35.296551 18264 tablet_bootstrap.cc:492] T 66121c347e6c443dac7c5123826a7ae1 P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:35.317363 18282 ts_tablet_manager.cc:902] T a3c4cbf779554d8b84a48de78a9c392e P 60ba3ffe3b7746469834cd91c84be14e: Initiating tablet copy from peer 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:35.318185 18282 tablet_copy_client.cc:308] T a3c4cbf779554d8b84a48de78a9c392e P 60ba3ffe3b7746469834cd91c84be14e: tablet copy: Beginning tablet copy session from remote peer at address 127.6.98.66:36387
I20240503 13:56:35.319110 13205 tablet_copy_service.cc:140] P 8ccf5a4ac450414c9d3579bd5800b770: Received BeginTabletCopySession request for tablet a3c4cbf779554d8b84a48de78a9c392e from peer 60ba3ffe3b7746469834cd91c84be14e ({username='slave'} at 127.6.98.65:53261)
I20240503 13:56:35.319234 13205 tablet_copy_service.cc:161] P 8ccf5a4ac450414c9d3579bd5800b770: Beginning new tablet copy session on tablet a3c4cbf779554d8b84a48de78a9c392e from peer 60ba3ffe3b7746469834cd91c84be14e at {username='slave'} at 127.6.98.65:53261: session id = 60ba3ffe3b7746469834cd91c84be14e-a3c4cbf779554d8b84a48de78a9c392e
I20240503 13:56:35.320681 13205 tablet_copy_source_session.cc:205] T a3c4cbf779554d8b84a48de78a9c392e P 8ccf5a4ac450414c9d3579bd5800b770: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:56:35.322322 18282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3c4cbf779554d8b84a48de78a9c392e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:35.324837 18282 tablet_copy_client.cc:783] T a3c4cbf779554d8b84a48de78a9c392e P 60ba3ffe3b7746469834cd91c84be14e: tablet copy: Starting download of 0 data blocks...
I20240503 13:56:35.325176 18264 tablet_bootstrap.cc:492] T 66121c347e6c443dac7c5123826a7ae1 P 61b887f0d40d497badeca5604c1f0546: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:56:35.325435 18264 tablet_bootstrap.cc:492] T 66121c347e6c443dac7c5123826a7ae1 P 61b887f0d40d497badeca5604c1f0546: Bootstrap complete.
I20240503 13:56:35.325660 18264 ts_tablet_manager.cc:1367] T 66121c347e6c443dac7c5123826a7ae1 P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.029s user 0.024s sys 0.001s
I20240503 13:56:35.326084 18264 raft_consensus.cc:348] T 66121c347e6c443dac7c5123826a7ae1 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:35.326184 18264 raft_consensus.cc:729] T 66121c347e6c443dac7c5123826a7ae1 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:35.326221 18282 tablet_copy_client.cc:647] T a3c4cbf779554d8b84a48de78a9c392e P 60ba3ffe3b7746469834cd91c84be14e: tablet copy: Starting download of 1 WAL segments...
I20240503 13:56:35.326336 18264 consensus_queue.cc:260] T 66121c347e6c443dac7c5123826a7ae1 P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:35.326709 18264 ts_tablet_manager.cc:1398] T 66121c347e6c443dac7c5123826a7ae1 P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:35.327301 13074 tablet_copy_service.cc:342] P 60ba3ffe3b7746469834cd91c84be14e: Request end of tablet copy session 61b887f0d40d497badeca5604c1f0546-66121c347e6c443dac7c5123826a7ae1 received from {username='slave'} at 127.6.98.69:58129
I20240503 13:56:35.327378 13074 tablet_copy_service.cc:433] P 60ba3ffe3b7746469834cd91c84be14e: ending tablet copy session 61b887f0d40d497badeca5604c1f0546-66121c347e6c443dac7c5123826a7ae1 on tablet 66121c347e6c443dac7c5123826a7ae1 with peer 61b887f0d40d497badeca5604c1f0546
I20240503 13:56:35.329210 18293 ts_tablet_manager.cc:902] T 4939335096b94444a61503841e2e81eb P 009f849ee7b54c80a3f5264695047e4d: Initiating tablet copy from peer ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:35.330964 18282 tablet_copy_client.cc:523] T a3c4cbf779554d8b84a48de78a9c392e P 60ba3ffe3b7746469834cd91c84be14e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:56:35.331559 18293 tablet_copy_client.cc:308] T 4939335096b94444a61503841e2e81eb P 009f849ee7b54c80a3f5264695047e4d: tablet copy: Beginning tablet copy session from remote peer at address 127.6.98.67:42279
I20240503 13:56:35.332131 13336 tablet_copy_service.cc:140] P ba0194098aa84449ae17e8a0d4c43703: Received BeginTabletCopySession request for tablet 4939335096b94444a61503841e2e81eb from peer 009f849ee7b54c80a3f5264695047e4d ({username='slave'} at 127.6.98.70:35625)
I20240503 13:56:35.332249 13336 tablet_copy_service.cc:161] P ba0194098aa84449ae17e8a0d4c43703: Beginning new tablet copy session on tablet 4939335096b94444a61503841e2e81eb from peer 009f849ee7b54c80a3f5264695047e4d at {username='slave'} at 127.6.98.70:35625: session id = 009f849ee7b54c80a3f5264695047e4d-4939335096b94444a61503841e2e81eb
I20240503 13:56:35.333000 18282 tablet_bootstrap.cc:492] T a3c4cbf779554d8b84a48de78a9c392e P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:35.333230 13336 tablet_copy_source_session.cc:205] T 4939335096b94444a61503841e2e81eb P ba0194098aa84449ae17e8a0d4c43703: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:56:35.333912 18298 ts_tablet_manager.cc:902] T 855c5fc4cefb48b58533f26d62595465 P 009f849ee7b54c80a3f5264695047e4d: Initiating tablet copy from peer 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:35.334165 18293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4939335096b94444a61503841e2e81eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:35.334506 18298 tablet_copy_client.cc:308] T 855c5fc4cefb48b58533f26d62595465 P 009f849ee7b54c80a3f5264695047e4d: tablet copy: Beginning tablet copy session from remote peer at address 127.6.98.65:34695
I20240503 13:56:35.334968 13074 tablet_copy_service.cc:140] P 60ba3ffe3b7746469834cd91c84be14e: Received BeginTabletCopySession request for tablet 855c5fc4cefb48b58533f26d62595465 from peer 009f849ee7b54c80a3f5264695047e4d ({username='slave'} at 127.6.98.70:47015)
I20240503 13:56:35.335443 13074 tablet_copy_service.cc:161] P 60ba3ffe3b7746469834cd91c84be14e: Beginning new tablet copy session on tablet 855c5fc4cefb48b58533f26d62595465 from peer 009f849ee7b54c80a3f5264695047e4d at {username='slave'} at 127.6.98.70:47015: session id = 009f849ee7b54c80a3f5264695047e4d-855c5fc4cefb48b58533f26d62595465
I20240503 13:56:35.336637 13074 tablet_copy_source_session.cc:205] T 855c5fc4cefb48b58533f26d62595465 P 60ba3ffe3b7746469834cd91c84be14e: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:56:35.336664 18293 tablet_copy_client.cc:783] T 4939335096b94444a61503841e2e81eb P 009f849ee7b54c80a3f5264695047e4d: tablet copy: Starting download of 0 data blocks...
I20240503 13:56:35.336788 18293 tablet_copy_client.cc:647] T 4939335096b94444a61503841e2e81eb P 009f849ee7b54c80a3f5264695047e4d: tablet copy: Starting download of 1 WAL segments...
I20240503 13:56:35.337687 18298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 855c5fc4cefb48b58533f26d62595465. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:35.337931 18293 tablet_copy_client.cc:523] T 4939335096b94444a61503841e2e81eb P 009f849ee7b54c80a3f5264695047e4d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:56:35.339768 18293 tablet_bootstrap.cc:492] T 4939335096b94444a61503841e2e81eb P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:35.341043 18324 ts_tablet_manager.cc:902] T 0f9224c2e3b946d993a5fa5c95a1d681 P 7690ee79577c45e3a2c8f8b552c0bff8: Initiating tablet copy from peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:35.341683 18298 tablet_copy_client.cc:783] T 855c5fc4cefb48b58533f26d62595465 P 009f849ee7b54c80a3f5264695047e4d: tablet copy: Starting download of 0 data blocks...
I20240503 13:56:35.341868 18324 tablet_copy_client.cc:308] T 0f9224c2e3b946d993a5fa5c95a1d681 P 7690ee79577c45e3a2c8f8b552c0bff8: tablet copy: Beginning tablet copy session from remote peer at address 127.6.98.72:44753
I20240503 13:56:35.342128 18298 tablet_copy_client.cc:647] T 855c5fc4cefb48b58533f26d62595465 P 009f849ee7b54c80a3f5264695047e4d: tablet copy: Starting download of 1 WAL segments...
I20240503 13:56:35.343864 18298 tablet_copy_client.cc:523] T 855c5fc4cefb48b58533f26d62595465 P 009f849ee7b54c80a3f5264695047e4d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:56:35.345283 18298 tablet_bootstrap.cc:492] T 855c5fc4cefb48b58533f26d62595465 P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:35.345922 18068 raft_consensus.cc:482] T 557f14aaf5564ebaa461720a4d7997ff P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:35.346032 18068 raft_consensus.cc:504] T 557f14aaf5564ebaa461720a4d7997ff P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:35.346524 18068 leader_election.cc:288] T 557f14aaf5564ebaa461720a4d7997ff P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951), ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:35.346875 15624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "557f14aaf5564ebaa461720a4d7997ff" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d" is_pre_election: true
I20240503 13:56:35.346876 13316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "557f14aaf5564ebaa461720a4d7997ff" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703" is_pre_election: true
I20240503 13:56:35.346990 15624 raft_consensus.cc:2454] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 0.
I20240503 13:56:35.346990 13316 raft_consensus.cc:2454] T 557f14aaf5564ebaa461720a4d7997ff P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 0.
I20240503 13:56:35.347286 15825 leader_election.cc:302] T 557f14aaf5564ebaa461720a4d7997ff P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba0194098aa84449ae17e8a0d4c43703, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:35.347452 18068 raft_consensus.cc:2781] T 557f14aaf5564ebaa461720a4d7997ff P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:35.347502 18068 raft_consensus.cc:482] T 557f14aaf5564ebaa461720a4d7997ff P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:35.347537 18068 raft_consensus.cc:3037] T 557f14aaf5564ebaa461720a4d7997ff P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:35.348527 18068 raft_consensus.cc:504] T 557f14aaf5564ebaa461720a4d7997ff P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:35.348867 18068 leader_election.cc:288] T 557f14aaf5564ebaa461720a4d7997ff P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 election: Requested vote from peers 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951), ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:35.349095 15624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "557f14aaf5564ebaa461720a4d7997ff" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d"
I20240503 13:56:35.349196 15624 raft_consensus.cc:3037] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:35.349386 13316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "557f14aaf5564ebaa461720a4d7997ff" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703"
I20240503 13:56:35.349476 13316 raft_consensus.cc:3037] T 557f14aaf5564ebaa461720a4d7997ff P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:35.350034 15624 raft_consensus.cc:2454] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 1.
I20240503 13:56:35.350404 13316 raft_consensus.cc:2454] T 557f14aaf5564ebaa461720a4d7997ff P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 1.
I20240503 13:56:35.350433 15822 leader_election.cc:302] T 557f14aaf5564ebaa461720a4d7997ff P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:35.350780 18068 raft_consensus.cc:2781] T 557f14aaf5564ebaa461720a4d7997ff P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:35.350869 18068 raft_consensus.cc:686] T 557f14aaf5564ebaa461720a4d7997ff P e47105a0f46e4511b7112e571688a4c7 [term 1 LEADER]: Becoming Leader. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Running, Role: LEADER
I20240503 13:56:35.350988 15909 tablet_copy_service.cc:140] P e47105a0f46e4511b7112e571688a4c7: Received BeginTabletCopySession request for tablet 0f9224c2e3b946d993a5fa5c95a1d681 from peer 7690ee79577c45e3a2c8f8b552c0bff8 ({username='slave'} at 127.6.98.68:38101)
I20240503 13:56:35.350996 18068 consensus_queue.cc:237] T 557f14aaf5564ebaa461720a4d7997ff P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:35.351106 15909 tablet_copy_service.cc:161] P e47105a0f46e4511b7112e571688a4c7: Beginning new tablet copy session on tablet 0f9224c2e3b946d993a5fa5c95a1d681 from peer 7690ee79577c45e3a2c8f8b552c0bff8 at {username='slave'} at 127.6.98.68:38101: session id = 7690ee79577c45e3a2c8f8b552c0bff8-0f9224c2e3b946d993a5fa5c95a1d681
I20240503 13:56:35.352330 15909 tablet_copy_source_session.cc:205] T 0f9224c2e3b946d993a5fa5c95a1d681 P e47105a0f46e4511b7112e571688a4c7: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:56:35.353847 18298 tablet_bootstrap.cc:492] T 855c5fc4cefb48b58533f26d62595465 P 009f849ee7b54c80a3f5264695047e4d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:56:35.354075 18298 tablet_bootstrap.cc:492] T 855c5fc4cefb48b58533f26d62595465 P 009f849ee7b54c80a3f5264695047e4d: Bootstrap complete.
I20240503 13:56:35.353606 12910 catalog_manager.cc:5484] T 557f14aaf5564ebaa461720a4d7997ff P e47105a0f46e4511b7112e571688a4c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e47105a0f46e4511b7112e571688a4c7 (127.6.98.72). New cstate: current_term: 1 leader_uuid: "e47105a0f46e4511b7112e571688a4c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:35.354226 18298 ts_tablet_manager.cc:1367] T 855c5fc4cefb48b58533f26d62595465 P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.002s
I20240503 13:56:35.355583 18068 raft_consensus.cc:482] T d3849e4effe64656bd8b0f504c9dc17a P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:35.355667 18068 raft_consensus.cc:504] T d3849e4effe64656bd8b0f504c9dc17a P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:35.356359 15624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3849e4effe64656bd8b0f504c9dc17a" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d" is_pre_election: true
I20240503 13:56:35.356433 18068 leader_election.cc:288] T d3849e4effe64656bd8b0f504c9dc17a P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:35.356472 15624 raft_consensus.cc:2454] T d3849e4effe64656bd8b0f504c9dc17a P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 0.
I20240503 13:56:35.356920 15822 leader_election.cc:302] T d3849e4effe64656bd8b0f504c9dc17a P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:35.357244 18068 raft_consensus.cc:2781] T d3849e4effe64656bd8b0f504c9dc17a P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:35.357327 18068 raft_consensus.cc:482] T d3849e4effe64656bd8b0f504c9dc17a P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:35.357390 18068 raft_consensus.cc:3037] T d3849e4effe64656bd8b0f504c9dc17a P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:35.357502 13183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3849e4effe64656bd8b0f504c9dc17a" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770" is_pre_election: true
I20240503 13:56:35.357623 13183 raft_consensus.cc:2454] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 0.
I20240503 13:56:35.358305 18068 raft_consensus.cc:504] T d3849e4effe64656bd8b0f504c9dc17a P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:35.358847 15624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3849e4effe64656bd8b0f504c9dc17a" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d"
I20240503 13:56:35.358954 15624 raft_consensus.cc:3037] T d3849e4effe64656bd8b0f504c9dc17a P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:35.359808 15624 raft_consensus.cc:2454] T d3849e4effe64656bd8b0f504c9dc17a P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 1.
I20240503 13:56:35.360118 15822 leader_election.cc:302] T d3849e4effe64656bd8b0f504c9dc17a P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:35.360353 18068 leader_election.cc:288] T d3849e4effe64656bd8b0f504c9dc17a P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 election: Requested vote from peers 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:35.360512 16404 raft_consensus.cc:2781] T d3849e4effe64656bd8b0f504c9dc17a P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:35.360605 16404 raft_consensus.cc:686] T d3849e4effe64656bd8b0f504c9dc17a P e47105a0f46e4511b7112e571688a4c7 [term 1 LEADER]: Becoming Leader. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Running, Role: LEADER
I20240503 13:56:35.360734 16404 consensus_queue.cc:237] T d3849e4effe64656bd8b0f504c9dc17a P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:35.361212 18324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f9224c2e3b946d993a5fa5c95a1d681. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:35.361284 18298 raft_consensus.cc:348] T 855c5fc4cefb48b58533f26d62595465 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:35.361824 18298 raft_consensus.cc:729] T 855c5fc4cefb48b58533f26d62595465 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:35.361961 18298 consensus_queue.cc:260] T 855c5fc4cefb48b58533f26d62595465 P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:35.362015 13183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3849e4effe64656bd8b0f504c9dc17a" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770"
I20240503 13:56:35.362129 13183 raft_consensus.cc:3037] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:35.362241 18298 ts_tablet_manager.cc:1398] T 855c5fc4cefb48b58533f26d62595465 P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240503 13:56:35.362726 13074 tablet_copy_service.cc:342] P 60ba3ffe3b7746469834cd91c84be14e: Request end of tablet copy session 009f849ee7b54c80a3f5264695047e4d-855c5fc4cefb48b58533f26d62595465 received from {username='slave'} at 127.6.98.70:47015
I20240503 13:56:35.362800 13074 tablet_copy_service.cc:433] P 60ba3ffe3b7746469834cd91c84be14e: ending tablet copy session 009f849ee7b54c80a3f5264695047e4d-855c5fc4cefb48b58533f26d62595465 on tablet 855c5fc4cefb48b58533f26d62595465 with peer 009f849ee7b54c80a3f5264695047e4d
I20240503 13:56:35.363112 13183 raft_consensus.cc:2454] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 1.
I20240503 13:56:35.363757 18324 tablet_copy_client.cc:783] T 0f9224c2e3b946d993a5fa5c95a1d681 P 7690ee79577c45e3a2c8f8b552c0bff8: tablet copy: Starting download of 0 data blocks...
I20240503 13:56:35.363780 12904 catalog_manager.cc:5484] T d3849e4effe64656bd8b0f504c9dc17a P e47105a0f46e4511b7112e571688a4c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e47105a0f46e4511b7112e571688a4c7 (127.6.98.72). New cstate: current_term: 1 leader_uuid: "e47105a0f46e4511b7112e571688a4c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } }
I20240503 13:56:35.363914 18324 tablet_copy_client.cc:647] T 0f9224c2e3b946d993a5fa5c95a1d681 P 7690ee79577c45e3a2c8f8b552c0bff8: tablet copy: Starting download of 1 WAL segments...
I20240503 13:56:35.365916 18282 tablet_bootstrap.cc:492] T a3c4cbf779554d8b84a48de78a9c392e P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:56:35.366205 18282 tablet_bootstrap.cc:492] T a3c4cbf779554d8b84a48de78a9c392e P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap complete.
I20240503 13:56:35.366364 18282 ts_tablet_manager.cc:1367] T a3c4cbf779554d8b84a48de78a9c392e P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.000s
I20240503 13:56:35.366823 18282 raft_consensus.cc:348] T a3c4cbf779554d8b84a48de78a9c392e P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:35.366951 18282 raft_consensus.cc:729] T a3c4cbf779554d8b84a48de78a9c392e P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:35.367089 18282 consensus_queue.cc:260] T a3c4cbf779554d8b84a48de78a9c392e P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:35.367380 18282 ts_tablet_manager.cc:1398] T a3c4cbf779554d8b84a48de78a9c392e P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 13:56:35.368680 13205 tablet_copy_service.cc:342] P 8ccf5a4ac450414c9d3579bd5800b770: Request end of tablet copy session 60ba3ffe3b7746469834cd91c84be14e-a3c4cbf779554d8b84a48de78a9c392e received from {username='slave'} at 127.6.98.65:53261
I20240503 13:56:35.368765 13205 tablet_copy_service.cc:433] P 8ccf5a4ac450414c9d3579bd5800b770: ending tablet copy session 60ba3ffe3b7746469834cd91c84be14e-a3c4cbf779554d8b84a48de78a9c392e on tablet a3c4cbf779554d8b84a48de78a9c392e with peer 60ba3ffe3b7746469834cd91c84be14e
I20240503 13:56:35.373070 18324 tablet_copy_client.cc:523] T 0f9224c2e3b946d993a5fa5c95a1d681 P 7690ee79577c45e3a2c8f8b552c0bff8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:56:35.374832 18324 tablet_bootstrap.cc:492] T 0f9224c2e3b946d993a5fa5c95a1d681 P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:35.379146 18324 tablet_bootstrap.cc:492] T 0f9224c2e3b946d993a5fa5c95a1d681 P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:56:35.379412 18324 tablet_bootstrap.cc:492] T 0f9224c2e3b946d993a5fa5c95a1d681 P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap complete.
I20240503 13:56:35.379581 18324 ts_tablet_manager.cc:1367] T 0f9224c2e3b946d993a5fa5c95a1d681 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20240503 13:56:35.380016 18324 raft_consensus.cc:348] T 0f9224c2e3b946d993a5fa5c95a1d681 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:35.380132 18324 raft_consensus.cc:729] T 0f9224c2e3b946d993a5fa5c95a1d681 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:35.380252 18324 consensus_queue.cc:260] T 0f9224c2e3b946d993a5fa5c95a1d681 P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:35.382607 18324 ts_tablet_manager.cc:1398] T 0f9224c2e3b946d993a5fa5c95a1d681 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:56:35.383154 15909 tablet_copy_service.cc:342] P e47105a0f46e4511b7112e571688a4c7: Request end of tablet copy session 7690ee79577c45e3a2c8f8b552c0bff8-0f9224c2e3b946d993a5fa5c95a1d681 received from {username='slave'} at 127.6.98.68:38101
I20240503 13:56:35.383250 15909 tablet_copy_service.cc:433] P e47105a0f46e4511b7112e571688a4c7: ending tablet copy session 7690ee79577c45e3a2c8f8b552c0bff8-0f9224c2e3b946d993a5fa5c95a1d681 on tablet 0f9224c2e3b946d993a5fa5c95a1d681 with peer 7690ee79577c45e3a2c8f8b552c0bff8
I20240503 13:56:35.388072 18293 tablet_bootstrap.cc:492] T 4939335096b94444a61503841e2e81eb P 009f849ee7b54c80a3f5264695047e4d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:56:35.388341 18293 tablet_bootstrap.cc:492] T 4939335096b94444a61503841e2e81eb P 009f849ee7b54c80a3f5264695047e4d: Bootstrap complete.
I20240503 13:56:35.388537 18293 ts_tablet_manager.cc:1367] T 4939335096b94444a61503841e2e81eb P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.049s user 0.023s sys 0.003s
I20240503 13:56:35.388985 18293 raft_consensus.cc:348] T 4939335096b94444a61503841e2e81eb P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:35.389117 18293 raft_consensus.cc:729] T 4939335096b94444a61503841e2e81eb P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:35.389266 18293 consensus_queue.cc:260] T 4939335096b94444a61503841e2e81eb P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:35.389689 18293 ts_tablet_manager.cc:1398] T 4939335096b94444a61503841e2e81eb P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:35.390157 13336 tablet_copy_service.cc:342] P ba0194098aa84449ae17e8a0d4c43703: Request end of tablet copy session 009f849ee7b54c80a3f5264695047e4d-4939335096b94444a61503841e2e81eb received from {username='slave'} at 127.6.98.70:35625
I20240503 13:56:35.390235 13336 tablet_copy_service.cc:433] P ba0194098aa84449ae17e8a0d4c43703: ending tablet copy session 009f849ee7b54c80a3f5264695047e4d-4939335096b94444a61503841e2e81eb on tablet 4939335096b94444a61503841e2e81eb with peer 009f849ee7b54c80a3f5264695047e4d
I20240503 13:56:35.393221 12909 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:35.393342 12909 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:35.395304 12909 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Sending DeleteTablet for 3 replicas of tablet a0bb459e3cfe4276833d30ca4e31bc90
I20240503 13:56:35.395731 15867 tablet_service.cc:1514] Processing DeleteTablet for tablet a0bb459e3cfe4276833d30ca4e31bc90 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:35 UTC) from {username='slave'} at 127.0.0.1:50498
I20240503 13:56:35.395891 15605 tablet_service.cc:1514] Processing DeleteTablet for tablet a0bb459e3cfe4276833d30ca4e31bc90 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:35 UTC) from {username='slave'} at 127.0.0.1:60718
I20240503 13:56:35.396163 18336 tablet_replica.cc:330] T a0bb459e3cfe4276833d30ca4e31bc90 P 009f849ee7b54c80a3f5264695047e4d: stopping tablet replica
I20240503 13:56:35.396346 18336 raft_consensus.cc:2229] T a0bb459e3cfe4276833d30ca4e31bc90 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:35.396459 15736 tablet_service.cc:1514] Processing DeleteTablet for tablet a0bb459e3cfe4276833d30ca4e31bc90 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:35 UTC) from {username='slave'} at 127.0.0.1:44984
I20240503 13:56:35.396584 18336 raft_consensus.cc:2258] T a0bb459e3cfe4276833d30ca4e31bc90 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:35.397305 18336 ts_tablet_manager.cc:1875] T a0bb459e3cfe4276833d30ca4e31bc90 P 009f849ee7b54c80a3f5264695047e4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:35.400548 18335 tablet_replica.cc:330] T a0bb459e3cfe4276833d30ca4e31bc90 P e47105a0f46e4511b7112e571688a4c7: stopping tablet replica
I20240503 13:56:35.400856 18336 ts_tablet_manager.cc:1888] T a0bb459e3cfe4276833d30ca4e31bc90 P 009f849ee7b54c80a3f5264695047e4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:35.401506 18336 log.cc:1196] T a0bb459e3cfe4276833d30ca4e31bc90 P 009f849ee7b54c80a3f5264695047e4d: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/wal/wals/a0bb459e3cfe4276833d30ca4e31bc90
I20240503 13:56:35.401851 18336 ts_tablet_manager.cc:1909] T a0bb459e3cfe4276833d30ca4e31bc90 P 009f849ee7b54c80a3f5264695047e4d: Deleting consensus metadata
I20240503 13:56:35.401933 18335 raft_consensus.cc:2229] T a0bb459e3cfe4276833d30ca4e31bc90 P e47105a0f46e4511b7112e571688a4c7 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:56:35.402272 18335 raft_consensus.cc:2258] T a0bb459e3cfe4276833d30ca4e31bc90 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:35.402604 12896 catalog_manager.cc:4830] TS 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951): tablet a0bb459e3cfe4276833d30ca4e31bc90 (table rebalancer_dynamic_table_new_name [id=36becc37cb0b4f4d93831bdbef7f85e5]) successfully deleted
I20240503 13:56:35.402824 18337 tablet_replica.cc:330] T a0bb459e3cfe4276833d30ca4e31bc90 P 76c386bef61a4bd5a907266a5118ffe2: stopping tablet replica
I20240503 13:56:35.402972 18337 raft_consensus.cc:2229] T a0bb459e3cfe4276833d30ca4e31bc90 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:35.404516 18335 ts_tablet_manager.cc:1875] T a0bb459e3cfe4276833d30ca4e31bc90 P e47105a0f46e4511b7112e571688a4c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:35.407282 18335 ts_tablet_manager.cc:1888] T a0bb459e3cfe4276833d30ca4e31bc90 P e47105a0f46e4511b7112e571688a4c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:35.407357 18335 log.cc:1196] T a0bb459e3cfe4276833d30ca4e31bc90 P e47105a0f46e4511b7112e571688a4c7: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-7/wal/wals/a0bb459e3cfe4276833d30ca4e31bc90
I20240503 13:56:35.407668 18335 ts_tablet_manager.cc:1909] T a0bb459e3cfe4276833d30ca4e31bc90 P e47105a0f46e4511b7112e571688a4c7: Deleting consensus metadata
I20240503 13:56:35.408315 12896 catalog_manager.cc:4830] TS e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): tablet a0bb459e3cfe4276833d30ca4e31bc90 (table rebalancer_dynamic_table_new_name [id=36becc37cb0b4f4d93831bdbef7f85e5]) successfully deleted
I20240503 13:56:35.408659 18337 raft_consensus.cc:2258] T a0bb459e3cfe4276833d30ca4e31bc90 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:35.409240 18337 ts_tablet_manager.cc:1875] T a0bb459e3cfe4276833d30ca4e31bc90 P 76c386bef61a4bd5a907266a5118ffe2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:35.411144 18337 ts_tablet_manager.cc:1888] T a0bb459e3cfe4276833d30ca4e31bc90 P 76c386bef61a4bd5a907266a5118ffe2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:35.411206 18337 log.cc:1196] T a0bb459e3cfe4276833d30ca4e31bc90 P 76c386bef61a4bd5a907266a5118ffe2: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/wal/wals/a0bb459e3cfe4276833d30ca4e31bc90
I20240503 13:56:35.411402 18337 ts_tablet_manager.cc:1909] T a0bb459e3cfe4276833d30ca4e31bc90 P 76c386bef61a4bd5a907266a5118ffe2: Deleting consensus metadata
I20240503 13:56:35.412218 12899 catalog_manager.cc:4830] TS 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885): tablet a0bb459e3cfe4276833d30ca4e31bc90 (table rebalancer_dynamic_table_new_name [id=36becc37cb0b4f4d93831bdbef7f85e5]) successfully deleted
I20240503 13:56:35.433131 18338 ts_tablet_manager.cc:902] T fdc5f76f63ec4528ad57202555caf243 P 8ccf5a4ac450414c9d3579bd5800b770: Initiating tablet copy from peer 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153)
I20240503 13:56:35.433828 18338 tablet_copy_client.cc:308] T fdc5f76f63ec4528ad57202555caf243 P 8ccf5a4ac450414c9d3579bd5800b770: tablet copy: Beginning tablet copy session from remote peer at address 127.6.98.69:39153
I20240503 13:56:35.433871 18186 raft_consensus.cc:482] T 63290f22e28b4329901dfc7ea6da0cc7 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:35.434015 18186 raft_consensus.cc:504] T 63290f22e28b4329901dfc7ea6da0cc7 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:35.434470 15513 tablet_copy_service.cc:140] P 61b887f0d40d497badeca5604c1f0546: Received BeginTabletCopySession request for tablet fdc5f76f63ec4528ad57202555caf243 from peer 8ccf5a4ac450414c9d3579bd5800b770 ({username='slave'} at 127.6.98.66:60977)
I20240503 13:56:35.434612 15513 tablet_copy_service.cc:161] P 61b887f0d40d497badeca5604c1f0546: Beginning new tablet copy session on tablet fdc5f76f63ec4528ad57202555caf243 from peer 8ccf5a4ac450414c9d3579bd5800b770 at {username='slave'} at 127.6.98.66:60977: session id = 8ccf5a4ac450414c9d3579bd5800b770-fdc5f76f63ec4528ad57202555caf243
I20240503 13:56:35.434615 18186 leader_election.cc:288] T 63290f22e28b4329901dfc7ea6da0cc7 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:35.434756 18051 raft_consensus.cc:482] T 63290f22e28b4329901dfc7ea6da0cc7 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:35.434839 18051 raft_consensus.cc:504] T 63290f22e28b4329901dfc7ea6da0cc7 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:35.434942 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63290f22e28b4329901dfc7ea6da0cc7" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2" is_pre_election: true
I20240503 13:56:35.435077 15757 raft_consensus.cc:2454] T 63290f22e28b4329901dfc7ea6da0cc7 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 0.
I20240503 13:56:35.435218 18051 leader_election.cc:288] T 63290f22e28b4329901dfc7ea6da0cc7 P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885), 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951)
I20240503 13:56:35.435348 15561 leader_election.cc:302] T 63290f22e28b4329901dfc7ea6da0cc7 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 76c386bef61a4bd5a907266a5118ffe2; no voters:
I20240503 13:56:35.435498 18186 raft_consensus.cc:2781] T 63290f22e28b4329901dfc7ea6da0cc7 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:35.435540 18186 raft_consensus.cc:482] T 63290f22e28b4329901dfc7ea6da0cc7 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:35.435571 18186 raft_consensus.cc:3037] T 63290f22e28b4329901dfc7ea6da0cc7 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:35.435552 15624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63290f22e28b4329901dfc7ea6da0cc7" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d" is_pre_election: true
I20240503 13:56:35.435796 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63290f22e28b4329901dfc7ea6da0cc7" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2" is_pre_election: true
I20240503 13:56:35.435892 15757 raft_consensus.cc:2454] T 63290f22e28b4329901dfc7ea6da0cc7 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 0.
I20240503 13:56:35.435990 15513 tablet_copy_source_session.cc:205] T fdc5f76f63ec4528ad57202555caf243 P 61b887f0d40d497badeca5604c1f0546: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:56:35.436261 13121 leader_election.cc:302] T 63290f22e28b4329901dfc7ea6da0cc7 P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76c386bef61a4bd5a907266a5118ffe2, 8ccf5a4ac450414c9d3579bd5800b770; no voters:
I20240503 13:56:35.436415 18051 raft_consensus.cc:2781] T 63290f22e28b4329901dfc7ea6da0cc7 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:35.436472 18051 raft_consensus.cc:482] T 63290f22e28b4329901dfc7ea6da0cc7 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:35.436542 18051 raft_consensus.cc:3037] T 63290f22e28b4329901dfc7ea6da0cc7 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:35.436666 18186 raft_consensus.cc:504] T 63290f22e28b4329901dfc7ea6da0cc7 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:35.436900 15624 raft_consensus.cc:2379] T 63290f22e28b4329901dfc7ea6da0cc7 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8ccf5a4ac450414c9d3579bd5800b770 in current term 1: Already voted for candidate 009f849ee7b54c80a3f5264695047e4d in this term.
I20240503 13:56:35.436993 18186 leader_election.cc:288] T 63290f22e28b4329901dfc7ea6da0cc7 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 election: Requested vote from peers 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:35.437292 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63290f22e28b4329901dfc7ea6da0cc7" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2"
I20240503 13:56:35.437415 15757 raft_consensus.cc:3037] T 63290f22e28b4329901dfc7ea6da0cc7 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:35.437551 18051 raft_consensus.cc:504] T 63290f22e28b4329901dfc7ea6da0cc7 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:35.437850 18051 leader_election.cc:288] T 63290f22e28b4329901dfc7ea6da0cc7 P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 election: Requested vote from peers 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885), 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951)
I20240503 13:56:35.438172 15624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63290f22e28b4329901dfc7ea6da0cc7" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d"
I20240503 13:56:35.438211 15757 raft_consensus.cc:2454] T 63290f22e28b4329901dfc7ea6da0cc7 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 1.
I20240503 13:56:35.438302 15624 raft_consensus.cc:2379] T 63290f22e28b4329901dfc7ea6da0cc7 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8ccf5a4ac450414c9d3579bd5800b770 in current term 1: Already voted for candidate 009f849ee7b54c80a3f5264695047e4d in this term.
I20240503 13:56:35.438601 15561 leader_election.cc:302] T 63290f22e28b4329901dfc7ea6da0cc7 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 76c386bef61a4bd5a907266a5118ffe2; no voters:
I20240503 13:56:35.438756 18186 raft_consensus.cc:2781] T 63290f22e28b4329901dfc7ea6da0cc7 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:35.438796 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63290f22e28b4329901dfc7ea6da0cc7" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2"
I20240503 13:56:35.438845 18186 raft_consensus.cc:686] T 63290f22e28b4329901dfc7ea6da0cc7 P 009f849ee7b54c80a3f5264695047e4d [term 1 LEADER]: Becoming Leader. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Running, Role: LEADER
I20240503 13:56:35.438918 15757 raft_consensus.cc:2379] T 63290f22e28b4329901dfc7ea6da0cc7 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8ccf5a4ac450414c9d3579bd5800b770 in current term 1: Already voted for candidate 009f849ee7b54c80a3f5264695047e4d in this term.
I20240503 13:56:35.438968 18186 consensus_queue.cc:237] T 63290f22e28b4329901dfc7ea6da0cc7 P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:35.439183 13121 leader_election.cc:302] T 63290f22e28b4329901dfc7ea6da0cc7 P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ccf5a4ac450414c9d3579bd5800b770; no voters: 009f849ee7b54c80a3f5264695047e4d, 76c386bef61a4bd5a907266a5118ffe2
I20240503 13:56:35.439324 18051 raft_consensus.cc:2726] T 63290f22e28b4329901dfc7ea6da0cc7 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 13:56:35.439656 18338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdc5f76f63ec4528ad57202555caf243. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:35.441030 12910 catalog_manager.cc:5484] T 63290f22e28b4329901dfc7ea6da0cc7 P 009f849ee7b54c80a3f5264695047e4d reported cstate change: term changed from 0 to 1, leader changed from <none> to 009f849ee7b54c80a3f5264695047e4d (127.6.98.70). New cstate: current_term: 1 leader_uuid: "009f849ee7b54c80a3f5264695047e4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } }
I20240503 13:56:35.441972 18338 tablet_copy_client.cc:783] T fdc5f76f63ec4528ad57202555caf243 P 8ccf5a4ac450414c9d3579bd5800b770: tablet copy: Starting download of 0 data blocks...
I20240503 13:56:35.442180 18338 tablet_copy_client.cc:647] T fdc5f76f63ec4528ad57202555caf243 P 8ccf5a4ac450414c9d3579bd5800b770: tablet copy: Starting download of 1 WAL segments...
I20240503 13:56:35.444278 18338 tablet_copy_client.cc:523] T fdc5f76f63ec4528ad57202555caf243 P 8ccf5a4ac450414c9d3579bd5800b770: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:56:35.444422 13183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63290f22e28b4329901dfc7ea6da0cc7" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770" is_pre_election: true
I20240503 13:56:35.444437 13185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63290f22e28b4329901dfc7ea6da0cc7" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770"
I20240503 13:56:35.444626 13183 raft_consensus.cc:2379] T 63290f22e28b4329901dfc7ea6da0cc7 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 009f849ee7b54c80a3f5264695047e4d in current term 1: Already voted for candidate 8ccf5a4ac450414c9d3579bd5800b770 in this term.
I20240503 13:56:35.445796 18338 tablet_bootstrap.cc:492] T fdc5f76f63ec4528ad57202555caf243 P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:35.449390 18338 tablet_bootstrap.cc:492] T fdc5f76f63ec4528ad57202555caf243 P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:56:35.449666 18338 tablet_bootstrap.cc:492] T fdc5f76f63ec4528ad57202555caf243 P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap complete.
I20240503 13:56:35.449826 18338 ts_tablet_manager.cc:1367] T fdc5f76f63ec4528ad57202555caf243 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20240503 13:56:35.450219 18338 raft_consensus.cc:348] T fdc5f76f63ec4528ad57202555caf243 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:35.450321 18338 raft_consensus.cc:729] T fdc5f76f63ec4528ad57202555caf243 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:35.450434 18338 consensus_queue.cc:260] T fdc5f76f63ec4528ad57202555caf243 P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:35.450711 18338 ts_tablet_manager.cc:1398] T fdc5f76f63ec4528ad57202555caf243 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:35.451329 15513 tablet_copy_service.cc:342] P 61b887f0d40d497badeca5604c1f0546: Request end of tablet copy session 8ccf5a4ac450414c9d3579bd5800b770-fdc5f76f63ec4528ad57202555caf243 received from {username='slave'} at 127.6.98.66:60977
I20240503 13:56:35.451419 15513 tablet_copy_service.cc:433] P 61b887f0d40d497badeca5604c1f0546: ending tablet copy session 8ccf5a4ac450414c9d3579bd5800b770-fdc5f76f63ec4528ad57202555caf243 on tablet fdc5f76f63ec4528ad57202555caf243 with peer 8ccf5a4ac450414c9d3579bd5800b770
I20240503 13:56:35.468751 15736 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:56:35.468770 15472 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:56:35.470105 13295 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:56:35.472230 15605 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:56:35.472470 13164 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:35.474412 13424 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:56:35.480665 13034 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:35.482954 15867 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:35.497684 12906 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:56:35.505106 15605 tablet_service.cc:1467] Processing CreateTablet for tablet 4746f41ea7cd4d3685534b79832fb567 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=59ea605de3724a4a8b0063298d924566]), partition=
I20240503 13:56:35.505388 15605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4746f41ea7cd4d3685534b79832fb567. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:35.507562 18253 tablet_bootstrap.cc:492] T 4746f41ea7cd4d3685534b79832fb567 P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:35.508864 18253 tablet_bootstrap.cc:654] T 4746f41ea7cd4d3685534b79832fb567 P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:35.509109 15736 tablet_service.cc:1467] Processing CreateTablet for tablet 4746f41ea7cd4d3685534b79832fb567 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=59ea605de3724a4a8b0063298d924566]), partition=
I20240503 13:56:35.509379 15736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4746f41ea7cd4d3685534b79832fb567. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:35.509979 15472 tablet_service.cc:1467] Processing CreateTablet for tablet 4746f41ea7cd4d3685534b79832fb567 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=59ea605de3724a4a8b0063298d924566]), partition=
I20240503 13:56:35.510202 15472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4746f41ea7cd4d3685534b79832fb567. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:35.511582 18256 tablet_bootstrap.cc:492] T 4746f41ea7cd4d3685534b79832fb567 P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:35.512454 18256 tablet_bootstrap.cc:654] T 4746f41ea7cd4d3685534b79832fb567 P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:35.514804 18360 tablet_bootstrap.cc:492] T 4746f41ea7cd4d3685534b79832fb567 P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:35.516052 18360 tablet_bootstrap.cc:654] T 4746f41ea7cd4d3685534b79832fb567 P 61b887f0d40d497badeca5604c1f0546: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:35.517432 18360 tablet_bootstrap.cc:492] T 4746f41ea7cd4d3685534b79832fb567 P 61b887f0d40d497badeca5604c1f0546: No bootstrap required, opened a new log
I20240503 13:56:35.517515 18360 ts_tablet_manager.cc:1367] T 4746f41ea7cd4d3685534b79832fb567 P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 13:56:35.517936 18360 raft_consensus.cc:348] T 4746f41ea7cd4d3685534b79832fb567 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:35.518050 18360 raft_consensus.cc:374] T 4746f41ea7cd4d3685534b79832fb567 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:35.518077 18360 raft_consensus.cc:729] T 4746f41ea7cd4d3685534b79832fb567 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:35.518186 18360 consensus_queue.cc:260] T 4746f41ea7cd4d3685534b79832fb567 P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:35.518452 18360 ts_tablet_manager.cc:1398] T 4746f41ea7cd4d3685534b79832fb567 P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:35.533798 18253 tablet_bootstrap.cc:492] T 4746f41ea7cd4d3685534b79832fb567 P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:35.533916 18253 ts_tablet_manager.cc:1367] T 4746f41ea7cd4d3685534b79832fb567 P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.000s
I20240503 13:56:35.534366 18253 raft_consensus.cc:348] T 4746f41ea7cd4d3685534b79832fb567 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:35.534490 18253 raft_consensus.cc:374] T 4746f41ea7cd4d3685534b79832fb567 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:35.534528 18253 raft_consensus.cc:729] T 4746f41ea7cd4d3685534b79832fb567 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:35.534657 18253 consensus_queue.cc:260] T 4746f41ea7cd4d3685534b79832fb567 P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:35.534948 18253 ts_tablet_manager.cc:1398] T 4746f41ea7cd4d3685534b79832fb567 P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:35.539803 18256 tablet_bootstrap.cc:492] T 4746f41ea7cd4d3685534b79832fb567 P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:35.539911 18256 ts_tablet_manager.cc:1367] T 4746f41ea7cd4d3685534b79832fb567 P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.028s user 0.001s sys 0.000s
I20240503 13:56:35.540378 18256 raft_consensus.cc:348] T 4746f41ea7cd4d3685534b79832fb567 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:35.540807 18256 raft_consensus.cc:374] T 4746f41ea7cd4d3685534b79832fb567 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:35.540846 18256 raft_consensus.cc:729] T 4746f41ea7cd4d3685534b79832fb567 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:35.540979 18256 consensus_queue.cc:260] T 4746f41ea7cd4d3685534b79832fb567 P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:35.541304 18256 ts_tablet_manager.cc:1398] T 4746f41ea7cd4d3685534b79832fb567 P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:35.611114 17858 raft_consensus.cc:482] T 4746f41ea7cd4d3685534b79832fb567 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:35.611253 17858 raft_consensus.cc:504] T 4746f41ea7cd4d3685534b79832fb567 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:35.611704 17858 leader_election.cc:288] T 4746f41ea7cd4d3685534b79832fb567 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885), 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951)
I20240503 13:56:35.612412 15624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4746f41ea7cd4d3685534b79832fb567" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d" is_pre_election: true
I20240503 13:56:35.612545 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4746f41ea7cd4d3685534b79832fb567" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2" is_pre_election: true
I20240503 13:56:35.612598 15624 raft_consensus.cc:2454] T 4746f41ea7cd4d3685534b79832fb567 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 0.
I20240503 13:56:35.612658 15757 raft_consensus.cc:2454] T 4746f41ea7cd4d3685534b79832fb567 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 0.
I20240503 13:56:35.612937 15426 leader_election.cc:302] T 4746f41ea7cd4d3685534b79832fb567 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 61b887f0d40d497badeca5604c1f0546; no voters:
I20240503 13:56:35.613363 17858 raft_consensus.cc:2781] T 4746f41ea7cd4d3685534b79832fb567 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:35.613426 17858 raft_consensus.cc:482] T 4746f41ea7cd4d3685534b79832fb567 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:35.613463 17858 raft_consensus.cc:3037] T 4746f41ea7cd4d3685534b79832fb567 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:35.614507 17858 raft_consensus.cc:504] T 4746f41ea7cd4d3685534b79832fb567 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:35.614874 17858 leader_election.cc:288] T 4746f41ea7cd4d3685534b79832fb567 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 election: Requested vote from peers 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885), 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951)
I20240503 13:56:35.615397 15624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4746f41ea7cd4d3685534b79832fb567" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d"
I20240503 13:56:35.615522 15624 raft_consensus.cc:3037] T 4746f41ea7cd4d3685534b79832fb567 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:35.615559 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4746f41ea7cd4d3685534b79832fb567" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2"
I20240503 13:56:35.615686 15757 raft_consensus.cc:3037] T 4746f41ea7cd4d3685534b79832fb567 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:35.616451 15624 raft_consensus.cc:2454] T 4746f41ea7cd4d3685534b79832fb567 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 1.
I20240503 13:56:35.616767 15426 leader_election.cc:302] T 4746f41ea7cd4d3685534b79832fb567 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 61b887f0d40d497badeca5604c1f0546; no voters:
I20240503 13:56:35.616912 17858 raft_consensus.cc:2781] T 4746f41ea7cd4d3685534b79832fb567 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:35.616991 17858 raft_consensus.cc:686] T 4746f41ea7cd4d3685534b79832fb567 P 61b887f0d40d497badeca5604c1f0546 [term 1 LEADER]: Becoming Leader. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Running, Role: LEADER
I20240503 13:56:35.617121 17858 consensus_queue.cc:237] T 4746f41ea7cd4d3685534b79832fb567 P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:35.619189 12905 catalog_manager.cc:5484] T 4746f41ea7cd4d3685534b79832fb567 P 61b887f0d40d497badeca5604c1f0546 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61b887f0d40d497badeca5604c1f0546 (127.6.98.69). New cstate: current_term: 1 leader_uuid: "61b887f0d40d497badeca5604c1f0546" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:35.624934 15757 raft_consensus.cc:2454] T 4746f41ea7cd4d3685534b79832fb567 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 1.
I20240503 13:56:35.645936 13034 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:35.647879 15472 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 13:56:35.651369 15867 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:35.651994 15736 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:56:35.652318 13164 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:35.658411 13424 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:56:35.667397 15605 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:56:35.676651 13295 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:56:35.688644 15493 raft_consensus.cc:1203] T 66121c347e6c443dac7c5123826a7ae1 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:56:35.742112 12905 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:56:35.754098 15624 raft_consensus.cc:1261] T 4746f41ea7cd4d3685534b79832fb567 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Refusing update from remote peer 61b887f0d40d497badeca5604c1f0546: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:35.754196 15757 raft_consensus.cc:1261] T 4746f41ea7cd4d3685534b79832fb567 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Refusing update from remote peer 61b887f0d40d497badeca5604c1f0546: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:35.754597 17385 consensus_queue.cc:1035] T 4746f41ea7cd4d3685534b79832fb567 P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:35.755167 17385 consensus_queue.cc:1035] T 4746f41ea7cd4d3685534b79832fb567 P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:35.757223 12905 catalog_manager.cc:5576] TS 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885) does not have the latest schema for tablet 4746f41ea7cd4d3685534b79832fb567 (table rebalancer_dynamic_table [id=59ea605de3724a4a8b0063298d924566]). Expected version 1 got 0
I20240503 13:56:35.758342 18442 tablet.cc:1722] T 4746f41ea7cd4d3685534b79832fb567 P 61b887f0d40d497badeca5604c1f0546: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:35.758770 12905 catalog_manager.cc:5576] TS 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951) does not have the latest schema for tablet 4746f41ea7cd4d3685534b79832fb567 (table rebalancer_dynamic_table [id=59ea605de3724a4a8b0063298d924566]). Expected version 1 got 0
I20240503 13:56:35.758787 18442 tablet.cc:1620] T 4746f41ea7cd4d3685534b79832fb567 P 61b887f0d40d497badeca5604c1f0546: MemRowSet was empty: no flush needed.
I20240503 13:56:35.759387 18195 tablet.cc:1722] T 4746f41ea7cd4d3685534b79832fb567 P 009f849ee7b54c80a3f5264695047e4d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:35.759689 18195 tablet.cc:1620] T 4746f41ea7cd4d3685534b79832fb567 P 009f849ee7b54c80a3f5264695047e4d: MemRowSet was empty: no flush needed.
I20240503 13:56:35.762832 12909 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=59ea605de3724a4a8b0063298d924566] alter complete (version 1)
I20240503 13:56:35.763141 18068 consensus_queue.cc:1035] T 557f14aaf5564ebaa461720a4d7997ff P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:35.763458 18193 tablet.cc:1722] T 4746f41ea7cd4d3685534b79832fb567 P 76c386bef61a4bd5a907266a5118ffe2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:35.763760 18193 tablet.cc:1620] T 4746f41ea7cd4d3685534b79832fb567 P 76c386bef61a4bd5a907266a5118ffe2: MemRowSet was empty: no flush needed.
I20240503 13:56:35.769208 17953 consensus_queue.cc:1035] T 557f14aaf5564ebaa461720a4d7997ff P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:35.772904 15624 raft_consensus.cc:1203] T 4939335096b94444a61503841e2e81eb P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:56:35.773562 16404 consensus_queue.cc:1035] T d3849e4effe64656bd8b0f504c9dc17a P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:35.781107 16404 consensus_queue.cc:1035] T d3849e4effe64656bd8b0f504c9dc17a P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:35.819993 13447 raft_consensus.cc:1203] T 0f9224c2e3b946d993a5fa5c95a1d681 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:56:35.822829 15625 raft_consensus.cc:1203] T 855c5fc4cefb48b58533f26d62595465 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:56:35.823221 15605 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:56:35.826470 13034 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:35.834741 13424 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:56:35.835865 15472 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 13:56:35.837458 15736 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:56:35.848856 15867 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:35.849547 13164 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:35.851239 13295 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:56:35.852576 13183 raft_consensus.cc:1203] T fdc5f76f63ec4528ad57202555caf243 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:56:35.865495 12911 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:56:35.871444 18442 tablet.cc:1722] T 4746f41ea7cd4d3685534b79832fb567 P 61b887f0d40d497badeca5604c1f0546: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:35.871798 18442 tablet.cc:1620] T 4746f41ea7cd4d3685534b79832fb567 P 61b887f0d40d497badeca5604c1f0546: MemRowSet was empty: no flush needed.
I20240503 13:56:35.871851 18193 tablet.cc:1722] T 4746f41ea7cd4d3685534b79832fb567 P 76c386bef61a4bd5a907266a5118ffe2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:35.872138 18193 tablet.cc:1620] T 4746f41ea7cd4d3685534b79832fb567 P 76c386bef61a4bd5a907266a5118ffe2: MemRowSet was empty: no flush needed.
I20240503 13:56:35.876801 18195 tablet.cc:1722] T 4746f41ea7cd4d3685534b79832fb567 P 009f849ee7b54c80a3f5264695047e4d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:35.877125 18195 tablet.cc:1620] T 4746f41ea7cd4d3685534b79832fb567 P 009f849ee7b54c80a3f5264695047e4d: MemRowSet was empty: no flush needed.
I20240503 13:56:35.877444 12910 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=59ea605de3724a4a8b0063298d924566] alter complete (version 2)
I20240503 13:56:35.913173 13051 raft_consensus.cc:1203] T a3c4cbf779554d8b84a48de78a9c392e P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:56:35.920112 15889 consensus_queue.cc:237] T 0f9224c2e3b946d993a5fa5c95a1d681 P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:35.921413 15757 raft_consensus.cc:1261] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Refusing update from remote peer e47105a0f46e4511b7112e571688a4c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:56:35.921442 13447 raft_consensus.cc:1261] T 0f9224c2e3b946d993a5fa5c95a1d681 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer e47105a0f46e4511b7112e571688a4c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:56:35.921829 17953 consensus_queue.cc:1035] T 0f9224c2e3b946d993a5fa5c95a1d681 P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:56:35.922326 17953 consensus_queue.cc:1035] T 0f9224c2e3b946d993a5fa5c95a1d681 P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:56:35.924710 16404 raft_consensus.cc:2932] T 0f9224c2e3b946d993a5fa5c95a1d681 P e47105a0f46e4511b7112e571688a4c7 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 009f849ee7b54c80a3f5264695047e4d (127.6.98.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } }
I20240503 13:56:35.925168 13447 raft_consensus.cc:2932] T 0f9224c2e3b946d993a5fa5c95a1d681 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 009f849ee7b54c80a3f5264695047e4d (127.6.98.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } }
I20240503 13:56:35.925362 15757 raft_consensus.cc:2932] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 009f849ee7b54c80a3f5264695047e4d (127.6.98.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } }
I20240503 13:56:35.928074 12910 catalog_manager.cc:5484] T 0f9224c2e3b946d993a5fa5c95a1d681 P e47105a0f46e4511b7112e571688a4c7 reported cstate change: config changed from index 2 to 3, VOTER 009f849ee7b54c80a3f5264695047e4d (127.6.98.70) evicted. New cstate: current_term: 1 leader_uuid: "e47105a0f46e4511b7112e571688a4c7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } }
I20240503 13:56:35.931097 15605 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f9224c2e3b946d993a5fa5c95a1d681 with delete_type TABLET_DATA_TOMBSTONED (TS 009f849ee7b54c80a3f5264695047e4d not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:60718
I20240503 13:56:35.932776 18499 tablet_replica.cc:330] T 0f9224c2e3b946d993a5fa5c95a1d681 P 009f849ee7b54c80a3f5264695047e4d: stopping tablet replica
I20240503 13:56:35.933007 18499 raft_consensus.cc:2229] T 0f9224c2e3b946d993a5fa5c95a1d681 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:35.933231 18499 raft_consensus.cc:2258] T 0f9224c2e3b946d993a5fa5c95a1d681 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:35.933584 18187 consensus_queue.cc:1035] T 63290f22e28b4329901dfc7ea6da0cc7 P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:35.933748 18499 ts_tablet_manager.cc:1875] T 0f9224c2e3b946d993a5fa5c95a1d681 P 009f849ee7b54c80a3f5264695047e4d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:56:35.936203 18499 ts_tablet_manager.cc:1888] T 0f9224c2e3b946d993a5fa5c95a1d681 P 009f849ee7b54c80a3f5264695047e4d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 13:56:35.936306 18499 log.cc:1196] T 0f9224c2e3b946d993a5fa5c95a1d681 P 009f849ee7b54c80a3f5264695047e4d: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/wal/wals/0f9224c2e3b946d993a5fa5c95a1d681
I20240503 13:56:35.936846 12896 catalog_manager.cc:4830] TS 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951): tablet 0f9224c2e3b946d993a5fa5c95a1d681 (table rebalancer_extra_table_2 [id=65fb8aeaab8a4eb884e47e4e0f7ef87d]) successfully deleted
I20240503 13:56:35.938208 18186 consensus_queue.cc:1035] T 63290f22e28b4329901dfc7ea6da0cc7 P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:35.981190 12909 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:56:35.988337 18442 tablet.cc:1722] T 4746f41ea7cd4d3685534b79832fb567 P 61b887f0d40d497badeca5604c1f0546: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:35.988798 18193 tablet.cc:1722] T 4746f41ea7cd4d3685534b79832fb567 P 76c386bef61a4bd5a907266a5118ffe2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:35.989434 18195 tablet.cc:1722] T 4746f41ea7cd4d3685534b79832fb567 P 009f849ee7b54c80a3f5264695047e4d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:35.995712 12907 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table_new_name [id=59ea605de3724a4a8b0063298d924566] alter complete (version 3)
I20240503 13:56:36.001336 12909 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_extra_table_18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:56:36.013906 13164 tablet_service.cc:1467] Processing CreateTablet for tablet ab1fed738e1c46a6a0a0f9e2b9cfd9b0 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=b35f421662a14e679ccbb3bc941e54f5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:36.014259 13164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab1fed738e1c46a6a0a0f9e2b9cfd9b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:36.014261 15736 tablet_service.cc:1467] Processing CreateTablet for tablet ab1fed738e1c46a6a0a0f9e2b9cfd9b0 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=b35f421662a14e679ccbb3bc941e54f5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:36.014559 15736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab1fed738e1c46a6a0a0f9e2b9cfd9b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:36.014261 15737 tablet_service.cc:1467] Processing CreateTablet for tablet af1af9a911c343e880441f2177383b25 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=b35f421662a14e679ccbb3bc941e54f5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:36.014835 15737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af1af9a911c343e880441f2177383b25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:36.015136 13295 tablet_service.cc:1467] Processing CreateTablet for tablet af1af9a911c343e880441f2177383b25 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=b35f421662a14e679ccbb3bc941e54f5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:36.015374 13295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af1af9a911c343e880441f2177383b25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:36.016539 18546 tablet_bootstrap.cc:492] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:36.016707 15738 tablet_service.cc:1467] Processing CreateTablet for tablet 10ee7dee14364389871900670a02906a (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=b35f421662a14e679ccbb3bc941e54f5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:36.016947 15738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10ee7dee14364389871900670a02906a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:36.017590 18546 tablet_bootstrap.cc:654] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:36.017817 18547 tablet_bootstrap.cc:492] T af1af9a911c343e880441f2177383b25 P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:36.018615 18547 tablet_bootstrap.cc:654] T af1af9a911c343e880441f2177383b25 P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:36.018944 18256 tablet_bootstrap.cc:492] T af1af9a911c343e880441f2177383b25 P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:36.019793 18256 tablet_bootstrap.cc:654] T af1af9a911c343e880441f2177383b25 P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:36.020571 15605 tablet_service.cc:1467] Processing CreateTablet for tablet 10ee7dee14364389871900670a02906a (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=b35f421662a14e679ccbb3bc941e54f5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:36.020675 15867 tablet_service.cc:1467] Processing CreateTablet for tablet ab1fed738e1c46a6a0a0f9e2b9cfd9b0 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=b35f421662a14e679ccbb3bc941e54f5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:36.020838 15605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10ee7dee14364389871900670a02906a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:36.020932 15867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab1fed738e1c46a6a0a0f9e2b9cfd9b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:36.021620 15604 tablet_service.cc:1467] Processing CreateTablet for tablet af1af9a911c343e880441f2177383b25 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=b35f421662a14e679ccbb3bc941e54f5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:36.021942 15604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af1af9a911c343e880441f2177383b25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:36.023020 18253 tablet_bootstrap.cc:492] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:36.023489 15868 tablet_service.cc:1467] Processing CreateTablet for tablet 10ee7dee14364389871900670a02906a (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=b35f421662a14e679ccbb3bc941e54f5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:36.023734 15868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10ee7dee14364389871900670a02906a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:36.023914 18253 tablet_bootstrap.cc:654] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:36.025487 18253 tablet_bootstrap.cc:492] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:36.025588 18253 ts_tablet_manager.cc:1367] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:56:36.025969 18253 raft_consensus.cc:348] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:36.026075 18253 raft_consensus.cc:374] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:36.026103 18253 raft_consensus.cc:729] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:36.026217 18253 consensus_queue.cc:260] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:36.026464 18253 ts_tablet_manager.cc:1398] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:36.026595 18253 tablet_bootstrap.cc:492] T af1af9a911c343e880441f2177383b25 P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:36.027009 18551 tablet_bootstrap.cc:492] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:36.027257 18256 tablet_bootstrap.cc:492] T af1af9a911c343e880441f2177383b25 P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:36.027351 18253 tablet_bootstrap.cc:654] T af1af9a911c343e880441f2177383b25 P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:36.027355 18256 ts_tablet_manager.cc:1367] T af1af9a911c343e880441f2177383b25 P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 13:56:36.027871 18551 tablet_bootstrap.cc:654] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P e47105a0f46e4511b7112e571688a4c7: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:36.027819 18256 raft_consensus.cc:348] T af1af9a911c343e880441f2177383b25 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:36.027938 18256 raft_consensus.cc:374] T af1af9a911c343e880441f2177383b25 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:36.027969 18256 raft_consensus.cc:729] T af1af9a911c343e880441f2177383b25 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:36.028102 18256 consensus_queue.cc:260] T af1af9a911c343e880441f2177383b25 P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:36.028256 18253 tablet_bootstrap.cc:492] T af1af9a911c343e880441f2177383b25 P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:36.028324 18253 ts_tablet_manager.cc:1367] T af1af9a911c343e880441f2177383b25 P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:36.028374 18256 ts_tablet_manager.cc:1398] T af1af9a911c343e880441f2177383b25 P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:36.028573 18256 tablet_bootstrap.cc:492] T 10ee7dee14364389871900670a02906a P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:36.028769 18253 raft_consensus.cc:348] T af1af9a911c343e880441f2177383b25 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:36.028856 18253 raft_consensus.cc:374] T af1af9a911c343e880441f2177383b25 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:36.028882 18253 raft_consensus.cc:729] T af1af9a911c343e880441f2177383b25 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:36.029017 18253 consensus_queue.cc:260] T af1af9a911c343e880441f2177383b25 P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:36.029300 18253 ts_tablet_manager.cc:1398] T af1af9a911c343e880441f2177383b25 P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:36.029409 18256 tablet_bootstrap.cc:654] T 10ee7dee14364389871900670a02906a P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:36.030329 18256 tablet_bootstrap.cc:492] T 10ee7dee14364389871900670a02906a P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:36.030411 18256 ts_tablet_manager.cc:1367] T 10ee7dee14364389871900670a02906a P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:36.030898 18256 raft_consensus.cc:348] T 10ee7dee14364389871900670a02906a P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:36.031069 18256 raft_consensus.cc:374] T 10ee7dee14364389871900670a02906a P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:36.031105 18256 raft_consensus.cc:729] T 10ee7dee14364389871900670a02906a P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:36.031258 18256 consensus_queue.cc:260] T 10ee7dee14364389871900670a02906a P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:36.031634 18256 ts_tablet_manager.cc:1398] T 10ee7dee14364389871900670a02906a P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:36.031877 18256 tablet_bootstrap.cc:492] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:36.033944 18256 tablet_bootstrap.cc:654] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:36.034623 18547 tablet_bootstrap.cc:492] T af1af9a911c343e880441f2177383b25 P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:36.034813 18547 ts_tablet_manager.cc:1367] T af1af9a911c343e880441f2177383b25 P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.002s
I20240503 13:56:36.036721 18256 tablet_bootstrap.cc:492] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:36.036818 18256 ts_tablet_manager.cc:1367] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:56:36.037256 18547 raft_consensus.cc:348] T af1af9a911c343e880441f2177383b25 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:36.037288 18256 raft_consensus.cc:348] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:36.037385 18547 raft_consensus.cc:374] T af1af9a911c343e880441f2177383b25 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:36.037397 18256 raft_consensus.cc:374] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:36.037422 18547 raft_consensus.cc:729] T af1af9a911c343e880441f2177383b25 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:36.037427 18256 raft_consensus.cc:729] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:36.037544 18256 consensus_queue.cc:260] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:36.037567 18547 consensus_queue.cc:260] T af1af9a911c343e880441f2177383b25 P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:36.037811 18256 ts_tablet_manager.cc:1398] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:36.038167 18546 tablet_bootstrap.cc:492] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:36.038429 18546 ts_tablet_manager.cc:1367] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20240503 13:56:36.038633 18547 ts_tablet_manager.cc:1398] T af1af9a911c343e880441f2177383b25 P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 13:56:36.042626 18546 raft_consensus.cc:348] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:36.043040 18546 raft_consensus.cc:374] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:36.043139 18546 raft_consensus.cc:729] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:36.043406 18546 consensus_queue.cc:260] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:36.043888 18546 ts_tablet_manager.cc:1398] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240503 13:56:36.044773 18551 tablet_bootstrap.cc:492] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P e47105a0f46e4511b7112e571688a4c7: No bootstrap required, opened a new log
I20240503 13:56:36.045003 18551 ts_tablet_manager.cc:1367] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.002s
I20240503 13:56:36.045598 18551 raft_consensus.cc:348] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:36.045778 18551 raft_consensus.cc:374] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:36.045851 18551 raft_consensus.cc:729] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:36.046039 18551 consensus_queue.cc:260] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:36.046469 18551 ts_tablet_manager.cc:1398] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:36.046737 18551 tablet_bootstrap.cc:492] T 10ee7dee14364389871900670a02906a P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:36.047835 15604 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:56:36.048254 13424 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:56:36.049384 18551 tablet_bootstrap.cc:654] T 10ee7dee14364389871900670a02906a P e47105a0f46e4511b7112e571688a4c7: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:36.053408 18551 tablet_bootstrap.cc:492] T 10ee7dee14364389871900670a02906a P e47105a0f46e4511b7112e571688a4c7: No bootstrap required, opened a new log
I20240503 13:56:36.053828 18551 ts_tablet_manager.cc:1367] T 10ee7dee14364389871900670a02906a P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240503 13:56:36.055720 18551 raft_consensus.cc:348] T 10ee7dee14364389871900670a02906a P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:36.055848 18551 raft_consensus.cc:374] T 10ee7dee14364389871900670a02906a P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:36.055876 18551 raft_consensus.cc:729] T 10ee7dee14364389871900670a02906a P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:36.056094 18551 consensus_queue.cc:260] T 10ee7dee14364389871900670a02906a P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:36.061265 18551 ts_tablet_manager.cc:1398] T 10ee7dee14364389871900670a02906a P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.007s user 0.000s sys 0.001s
I20240503 13:56:36.065483 13034 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:36.069233 13164 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:36.072423 15736 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:56:36.073659 15472 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 13:56:36.076323 15868 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:36.079443 17953 raft_consensus.cc:482] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:36.079542 17953 raft_consensus.cc:504] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:36.079903 17953 leader_election.cc:288] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:36.084023 13295 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:56:36.084636 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1fed738e1c46a6a0a0f9e2b9cfd9b0" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2" is_pre_election: true
I20240503 13:56:36.084816 15757 raft_consensus.cc:2454] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 0.
I20240503 13:56:36.084940 13183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1fed738e1c46a6a0a0f9e2b9cfd9b0" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770" is_pre_election: true
I20240503 13:56:36.085070 13183 raft_consensus.cc:2454] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 0.
I20240503 13:56:36.088249 15825 leader_election.cc:302] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ccf5a4ac450414c9d3579bd5800b770, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:36.088599 17953 raft_consensus.cc:2781] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:36.088661 17953 raft_consensus.cc:482] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:36.088696 17953 raft_consensus.cc:3037] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:36.089736 17953 raft_consensus.cc:504] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:36.090077 17953 leader_election.cc:288] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 election: Requested vote from peers 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:36.090723 13183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1fed738e1c46a6a0a0f9e2b9cfd9b0" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770"
I20240503 13:56:36.090849 13183 raft_consensus.cc:3037] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:36.091858 13183 raft_consensus.cc:2454] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 1.
I20240503 13:56:36.092116 15825 leader_election.cc:302] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ccf5a4ac450414c9d3579bd5800b770, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:36.092280 17953 raft_consensus.cc:2781] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:36.092378 17953 raft_consensus.cc:686] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P e47105a0f46e4511b7112e571688a4c7 [term 1 LEADER]: Becoming Leader. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Running, Role: LEADER
I20240503 13:56:36.092547 17953 consensus_queue.cc:237] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:36.093298 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1fed738e1c46a6a0a0f9e2b9cfd9b0" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2"
I20240503 13:56:36.093422 15757 raft_consensus.cc:3037] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:36.094457 15757 raft_consensus.cc:2454] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 1.
I20240503 13:56:36.094875 12907 catalog_manager.cc:5484] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P e47105a0f46e4511b7112e571688a4c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e47105a0f46e4511b7112e571688a4c7 (127.6.98.72). New cstate: current_term: 1 leader_uuid: "e47105a0f46e4511b7112e571688a4c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } }
I20240503 13:56:36.101573 12906 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:36.101730 12906 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:36.103541 18186 raft_consensus.cc:482] T af1af9a911c343e880441f2177383b25 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:36.103646 18186 raft_consensus.cc:504] T af1af9a911c343e880441f2177383b25 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:36.103988 18186 leader_election.cc:288] T af1af9a911c343e880441f2177383b25 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885), ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:36.103992 12906 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Sending DeleteTablet for 3 replicas of tablet 4746f41ea7cd4d3685534b79832fb567
I20240503 13:56:36.104566 15604 tablet_service.cc:1514] Processing DeleteTablet for tablet 4746f41ea7cd4d3685534b79832fb567 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:36 UTC) from {username='slave'} at 127.0.0.1:60718
I20240503 13:56:36.104743 13314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1af9a911c343e880441f2177383b25" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703" is_pre_election: true
I20240503 13:56:36.104794 15472 tablet_service.cc:1514] Processing DeleteTablet for tablet 4746f41ea7cd4d3685534b79832fb567 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:36 UTC) from {username='slave'} at 127.0.0.1:58758
I20240503 13:56:36.104869 13314 raft_consensus.cc:2454] T af1af9a911c343e880441f2177383b25 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 0.
I20240503 13:56:36.105058 18499 tablet_replica.cc:330] T 4746f41ea7cd4d3685534b79832fb567 P 009f849ee7b54c80a3f5264695047e4d: stopping tablet replica
I20240503 13:56:36.105185 18499 raft_consensus.cc:2229] T 4746f41ea7cd4d3685534b79832fb567 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:36.105190 15736 tablet_service.cc:1514] Processing DeleteTablet for tablet 4746f41ea7cd4d3685534b79832fb567 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:36 UTC) from {username='slave'} at 127.0.0.1:44984
I20240503 13:56:36.105283 18499 raft_consensus.cc:2258] T 4746f41ea7cd4d3685534b79832fb567 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:36.105999 18499 ts_tablet_manager.cc:1875] T 4746f41ea7cd4d3685534b79832fb567 P 009f849ee7b54c80a3f5264695047e4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:36.106861 17953 raft_consensus.cc:482] T 10ee7dee14364389871900670a02906a P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:36.106949 17953 raft_consensus.cc:504] T 10ee7dee14364389871900670a02906a P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:36.107839 17953 leader_election.cc:288] T 10ee7dee14364389871900670a02906a P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951), 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885)
I20240503 13:56:36.107834 15625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10ee7dee14364389871900670a02906a" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d" is_pre_election: true
I20240503 13:56:36.107967 15625 raft_consensus.cc:2454] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 0.
I20240503 13:56:36.108260 15758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1af9a911c343e880441f2177383b25" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2" is_pre_election: true
I20240503 13:56:36.108397 15758 raft_consensus.cc:2454] T af1af9a911c343e880441f2177383b25 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 0.
I20240503 13:56:36.108398 15822 leader_election.cc:302] T 10ee7dee14364389871900670a02906a P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:36.108628 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10ee7dee14364389871900670a02906a" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2" is_pre_election: true
I20240503 13:56:36.108728 15757 raft_consensus.cc:2454] T 10ee7dee14364389871900670a02906a P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 0.
I20240503 13:56:36.109081 17953 raft_consensus.cc:2781] T 10ee7dee14364389871900670a02906a P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:36.109135 17953 raft_consensus.cc:482] T 10ee7dee14364389871900670a02906a P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:36.109128 15561 leader_election.cc:302] T af1af9a911c343e880441f2177383b25 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 76c386bef61a4bd5a907266a5118ffe2; no voters:
I20240503 13:56:36.109171 17953 raft_consensus.cc:3037] T 10ee7dee14364389871900670a02906a P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:36.109452 18186 raft_consensus.cc:2781] T af1af9a911c343e880441f2177383b25 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:36.109500 18186 raft_consensus.cc:482] T af1af9a911c343e880441f2177383b25 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:36.109531 18186 raft_consensus.cc:3037] T af1af9a911c343e880441f2177383b25 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:36.109867 18571 tablet_replica.cc:330] T 4746f41ea7cd4d3685534b79832fb567 P 76c386bef61a4bd5a907266a5118ffe2: stopping tablet replica
I20240503 13:56:36.110014 18571 raft_consensus.cc:2229] T 4746f41ea7cd4d3685534b79832fb567 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:36.110121 18571 raft_consensus.cc:2258] T 4746f41ea7cd4d3685534b79832fb567 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:36.110170 17953 raft_consensus.cc:504] T 10ee7dee14364389871900670a02906a P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:36.110381 18570 tablet_replica.cc:330] T 4746f41ea7cd4d3685534b79832fb567 P 61b887f0d40d497badeca5604c1f0546: stopping tablet replica
I20240503 13:56:36.110375 18186 raft_consensus.cc:504] T af1af9a911c343e880441f2177383b25 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:36.110502 18570 raft_consensus.cc:2229] T 4746f41ea7cd4d3685534b79832fb567 P 61b887f0d40d497badeca5604c1f0546 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:56:36.110688 18570 raft_consensus.cc:2258] T 4746f41ea7cd4d3685534b79832fb567 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:36.110693 18186 leader_election.cc:288] T af1af9a911c343e880441f2177383b25 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 election: Requested vote from peers 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885), ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:36.111060 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10ee7dee14364389871900670a02906a" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2"
I20240503 13:56:36.111100 15625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10ee7dee14364389871900670a02906a" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d"
I20240503 13:56:36.111160 15757 raft_consensus.cc:3037] T 10ee7dee14364389871900670a02906a P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:36.111182 15625 raft_consensus.cc:3037] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:36.111253 18570 ts_tablet_manager.cc:1875] T 4746f41ea7cd4d3685534b79832fb567 P 61b887f0d40d497badeca5604c1f0546: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:36.111970 15625 raft_consensus.cc:2454] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 1.
I20240503 13:56:36.112023 15757 raft_consensus.cc:2454] T 10ee7dee14364389871900670a02906a P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 1.
I20240503 13:56:36.112277 15822 leader_election.cc:302] T 10ee7dee14364389871900670a02906a P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:36.112437 16404 raft_consensus.cc:2781] T 10ee7dee14364389871900670a02906a P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:36.112560 16404 raft_consensus.cc:686] T 10ee7dee14364389871900670a02906a P e47105a0f46e4511b7112e571688a4c7 [term 1 LEADER]: Becoming Leader. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Running, Role: LEADER
I20240503 13:56:36.112694 16404 consensus_queue.cc:237] T 10ee7dee14364389871900670a02906a P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:36.112812 13314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1af9a911c343e880441f2177383b25" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703"
I20240503 13:56:36.112916 13314 raft_consensus.cc:3037] T af1af9a911c343e880441f2177383b25 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:36.113379 18570 ts_tablet_manager.cc:1888] T 4746f41ea7cd4d3685534b79832fb567 P 61b887f0d40d497badeca5604c1f0546: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:36.113438 18570 log.cc:1196] T 4746f41ea7cd4d3685534b79832fb567 P 61b887f0d40d497badeca5604c1f0546: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/wal/wals/4746f41ea7cd4d3685534b79832fb567
I20240503 13:56:36.113658 18570 ts_tablet_manager.cc:1909] T 4746f41ea7cd4d3685534b79832fb567 P 61b887f0d40d497badeca5604c1f0546: Deleting consensus metadata
I20240503 13:56:36.113915 13314 raft_consensus.cc:2454] T af1af9a911c343e880441f2177383b25 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 1.
I20240503 13:56:36.114600 15758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1af9a911c343e880441f2177383b25" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2"
I20240503 13:56:36.114708 15758 raft_consensus.cc:3037] T af1af9a911c343e880441f2177383b25 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:36.114881 17953 leader_election.cc:288] T 10ee7dee14364389871900670a02906a P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 election: Requested vote from peers 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951), 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885)
I20240503 13:56:36.114956 12906 catalog_manager.cc:5484] T 10ee7dee14364389871900670a02906a P e47105a0f46e4511b7112e571688a4c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e47105a0f46e4511b7112e571688a4c7 (127.6.98.72). New cstate: current_term: 1 leader_uuid: "e47105a0f46e4511b7112e571688a4c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:36.115541 12899 catalog_manager.cc:4830] TS 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153): tablet 4746f41ea7cd4d3685534b79832fb567 (table rebalancer_dynamic_table_new_name [id=59ea605de3724a4a8b0063298d924566]) successfully deleted
I20240503 13:56:36.115638 15758 raft_consensus.cc:2454] T af1af9a911c343e880441f2177383b25 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 1.
I20240503 13:56:36.116436 15561 leader_election.cc:302] T af1af9a911c343e880441f2177383b25 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 76c386bef61a4bd5a907266a5118ffe2; no voters:
I20240503 13:56:36.116783 18186 raft_consensus.cc:2781] T af1af9a911c343e880441f2177383b25 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:36.116856 18186 raft_consensus.cc:686] T af1af9a911c343e880441f2177383b25 P 009f849ee7b54c80a3f5264695047e4d [term 1 LEADER]: Becoming Leader. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Running, Role: LEADER
I20240503 13:56:36.116966 18186 consensus_queue.cc:237] T af1af9a911c343e880441f2177383b25 P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:36.118971 18499 ts_tablet_manager.cc:1888] T 4746f41ea7cd4d3685534b79832fb567 P 009f849ee7b54c80a3f5264695047e4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:36.118914 12906 catalog_manager.cc:5484] T af1af9a911c343e880441f2177383b25 P 009f849ee7b54c80a3f5264695047e4d reported cstate change: term changed from 0 to 1, leader changed from <none> to 009f849ee7b54c80a3f5264695047e4d (127.6.98.70). New cstate: current_term: 1 leader_uuid: "009f849ee7b54c80a3f5264695047e4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:36.119055 18499 log.cc:1196] T 4746f41ea7cd4d3685534b79832fb567 P 009f849ee7b54c80a3f5264695047e4d: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/wal/wals/4746f41ea7cd4d3685534b79832fb567
I20240503 13:56:36.119308 18499 ts_tablet_manager.cc:1909] T 4746f41ea7cd4d3685534b79832fb567 P 009f849ee7b54c80a3f5264695047e4d: Deleting consensus metadata
I20240503 13:56:36.120291 12896 catalog_manager.cc:4830] TS 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951): tablet 4746f41ea7cd4d3685534b79832fb567 (table rebalancer_dynamic_table_new_name [id=59ea605de3724a4a8b0063298d924566]) successfully deleted
I20240503 13:56:36.125047 18571 ts_tablet_manager.cc:1875] T 4746f41ea7cd4d3685534b79832fb567 P 76c386bef61a4bd5a907266a5118ffe2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:36.127040 18571 ts_tablet_manager.cc:1888] T 4746f41ea7cd4d3685534b79832fb567 P 76c386bef61a4bd5a907266a5118ffe2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:36.127218 18571 log.cc:1196] T 4746f41ea7cd4d3685534b79832fb567 P 76c386bef61a4bd5a907266a5118ffe2: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/wal/wals/4746f41ea7cd4d3685534b79832fb567
I20240503 13:56:36.127535 18571 ts_tablet_manager.cc:1909] T 4746f41ea7cd4d3685534b79832fb567 P 76c386bef61a4bd5a907266a5118ffe2: Deleting consensus metadata
I20240503 13:56:36.128284 12899 catalog_manager.cc:4830] TS 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885): tablet 4746f41ea7cd4d3685534b79832fb567 (table rebalancer_dynamic_table_new_name [id=59ea605de3724a4a8b0063298d924566]) successfully deleted
I20240503 13:56:36.160823 13314 consensus_queue.cc:237] T 4939335096b94444a61503841e2e81eb P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:36.162101 13183 raft_consensus.cc:1261] T 4939335096b94444a61503841e2e81eb P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:56:36.162148 15625 raft_consensus.cc:1261] T 4939335096b94444a61503841e2e81eb P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:56:36.162600 18025 consensus_queue.cc:1035] T 4939335096b94444a61503841e2e81eb P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:36.162737 18031 consensus_queue.cc:1035] T 4939335096b94444a61503841e2e81eb P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:36.164610 18592 raft_consensus.cc:2932] T 4939335096b94444a61503841e2e81eb P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:36.164891 15625 raft_consensus.cc:2932] T 4939335096b94444a61503841e2e81eb P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:36.164921 13183 raft_consensus.cc:2932] T 4939335096b94444a61503841e2e81eb P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:36.167500 12906 catalog_manager.cc:5484] T 4939335096b94444a61503841e2e81eb P 009f849ee7b54c80a3f5264695047e4d reported cstate change: config changed from index 4 to 5, VOTER 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65) evicted. New cstate: current_term: 1 leader_uuid: "ba0194098aa84449ae17e8a0d4c43703" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:36.170074 13034 tablet_service.cc:1514] Processing DeleteTablet for tablet 4939335096b94444a61503841e2e81eb with delete_type TABLET_DATA_TOMBSTONED (TS 60ba3ffe3b7746469834cd91c84be14e not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:59458
I20240503 13:56:36.170430 18593 tablet_replica.cc:330] T 4939335096b94444a61503841e2e81eb P 60ba3ffe3b7746469834cd91c84be14e: stopping tablet replica
I20240503 13:56:36.170635 18593 raft_consensus.cc:2229] T 4939335096b94444a61503841e2e81eb P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:36.170831 18593 raft_consensus.cc:2258] T 4939335096b94444a61503841e2e81eb P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:36.171329 18593 ts_tablet_manager.cc:1875] T 4939335096b94444a61503841e2e81eb P 60ba3ffe3b7746469834cd91c84be14e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:56:36.173547 18593 ts_tablet_manager.cc:1888] T 4939335096b94444a61503841e2e81eb P 60ba3ffe3b7746469834cd91c84be14e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 13:56:36.173614 18593 log.cc:1196] T 4939335096b94444a61503841e2e81eb P 60ba3ffe3b7746469834cd91c84be14e: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal/wals/4939335096b94444a61503841e2e81eb
I20240503 13:56:36.174036 12897 catalog_manager.cc:4830] TS 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695): tablet 4939335096b94444a61503841e2e81eb (table rebalance_test_table_15 [id=2bf18d88eb674480901ad018abd82ad1]) successfully deleted
I20240503 13:56:36.205664 12909 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:56:36.211519 15472 tablet_service.cc:1467] Processing CreateTablet for tablet a8ce0b34487a4646b717c61f5e9d9e91 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=68ff44ab080b4bbb91c25aa9b06de4d5]), partition=
I20240503 13:56:36.211814 15472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8ce0b34487a4646b717c61f5e9d9e91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:36.211839 13424 tablet_service.cc:1467] Processing CreateTablet for tablet a8ce0b34487a4646b717c61f5e9d9e91 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=68ff44ab080b4bbb91c25aa9b06de4d5]), partition=
I20240503 13:56:36.211864 15736 tablet_service.cc:1467] Processing CreateTablet for tablet a8ce0b34487a4646b717c61f5e9d9e91 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=68ff44ab080b4bbb91c25aa9b06de4d5]), partition=
I20240503 13:56:36.212050 13424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8ce0b34487a4646b717c61f5e9d9e91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:36.212126 15736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8ce0b34487a4646b717c61f5e9d9e91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:36.215431 18256 tablet_bootstrap.cc:492] T a8ce0b34487a4646b717c61f5e9d9e91 P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:36.216373 18256 tablet_bootstrap.cc:654] T a8ce0b34487a4646b717c61f5e9d9e91 P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:36.217584 18256 tablet_bootstrap.cc:492] T a8ce0b34487a4646b717c61f5e9d9e91 P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:36.217677 18256 ts_tablet_manager.cc:1367] T a8ce0b34487a4646b717c61f5e9d9e91 P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:56:36.218050 18256 raft_consensus.cc:348] T a8ce0b34487a4646b717c61f5e9d9e91 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:36.218138 18256 raft_consensus.cc:374] T a8ce0b34487a4646b717c61f5e9d9e91 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:36.218165 18256 raft_consensus.cc:729] T a8ce0b34487a4646b717c61f5e9d9e91 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:36.218292 18256 consensus_queue.cc:260] T a8ce0b34487a4646b717c61f5e9d9e91 P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:36.218585 18256 ts_tablet_manager.cc:1398] T a8ce0b34487a4646b717c61f5e9d9e91 P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:36.220705 18616 tablet_bootstrap.cc:492] T a8ce0b34487a4646b717c61f5e9d9e91 P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:36.221570 18616 tablet_bootstrap.cc:654] T a8ce0b34487a4646b717c61f5e9d9e91 P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:36.222406 18615 tablet_bootstrap.cc:492] T a8ce0b34487a4646b717c61f5e9d9e91 P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:36.223372 18615 tablet_bootstrap.cc:654] T a8ce0b34487a4646b717c61f5e9d9e91 P 61b887f0d40d497badeca5604c1f0546: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:36.224608 18615 tablet_bootstrap.cc:492] T a8ce0b34487a4646b717c61f5e9d9e91 P 61b887f0d40d497badeca5604c1f0546: No bootstrap required, opened a new log
I20240503 13:56:36.224697 18615 ts_tablet_manager.cc:1367] T a8ce0b34487a4646b717c61f5e9d9e91 P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:56:36.225138 18615 raft_consensus.cc:348] T a8ce0b34487a4646b717c61f5e9d9e91 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:36.225273 18615 raft_consensus.cc:374] T a8ce0b34487a4646b717c61f5e9d9e91 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:36.225317 18615 raft_consensus.cc:729] T a8ce0b34487a4646b717c61f5e9d9e91 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:36.225440 18615 consensus_queue.cc:260] T a8ce0b34487a4646b717c61f5e9d9e91 P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:36.225713 18615 ts_tablet_manager.cc:1398] T a8ce0b34487a4646b717c61f5e9d9e91 P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:36.234813 18616 tablet_bootstrap.cc:492] T a8ce0b34487a4646b717c61f5e9d9e91 P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:36.234925 18616 ts_tablet_manager.cc:1367] T a8ce0b34487a4646b717c61f5e9d9e91 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240503 13:56:36.235512 18616 raft_consensus.cc:348] T a8ce0b34487a4646b717c61f5e9d9e91 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:36.235615 18616 raft_consensus.cc:374] T a8ce0b34487a4646b717c61f5e9d9e91 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:36.235643 18616 raft_consensus.cc:729] T a8ce0b34487a4646b717c61f5e9d9e91 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:36.235764 18616 consensus_queue.cc:260] T a8ce0b34487a4646b717c61f5e9d9e91 P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:36.236037 18616 ts_tablet_manager.cc:1398] T a8ce0b34487a4646b717c61f5e9d9e91 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:36.335312 18197 raft_consensus.cc:482] T a8ce0b34487a4646b717c61f5e9d9e91 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:36.335443 18197 raft_consensus.cc:504] T a8ce0b34487a4646b717c61f5e9d9e91 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:36.335829 18197 leader_election.cc:288] T a8ce0b34487a4646b717c61f5e9d9e91 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153), 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:36.336215 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8ce0b34487a4646b717c61f5e9d9e91" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
I20240503 13:56:36.336270 15493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8ce0b34487a4646b717c61f5e9d9e91" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546" is_pre_election: true
I20240503 13:56:36.336359 13447 raft_consensus.cc:2454] T a8ce0b34487a4646b717c61f5e9d9e91 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 0.
I20240503 13:56:36.336412 15493 raft_consensus.cc:2454] T a8ce0b34487a4646b717c61f5e9d9e91 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 0.
I20240503 13:56:36.336723 15694 leader_election.cc:302] T a8ce0b34487a4646b717c61f5e9d9e91 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61b887f0d40d497badeca5604c1f0546, 76c386bef61a4bd5a907266a5118ffe2; no voters:
I20240503 13:56:36.336925 18197 raft_consensus.cc:2781] T a8ce0b34487a4646b717c61f5e9d9e91 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:36.336983 18197 raft_consensus.cc:482] T a8ce0b34487a4646b717c61f5e9d9e91 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:36.337034 18197 raft_consensus.cc:3037] T a8ce0b34487a4646b717c61f5e9d9e91 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:36.338264 18197 raft_consensus.cc:504] T a8ce0b34487a4646b717c61f5e9d9e91 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:36.338676 18197 leader_election.cc:288] T a8ce0b34487a4646b717c61f5e9d9e91 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 election: Requested vote from peers 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153), 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:36.339011 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8ce0b34487a4646b717c61f5e9d9e91" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
I20240503 13:56:36.339037 15493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8ce0b34487a4646b717c61f5e9d9e91" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546"
I20240503 13:56:36.339118 13447 raft_consensus.cc:3037] T a8ce0b34487a4646b717c61f5e9d9e91 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:36.339148 15493 raft_consensus.cc:3037] T a8ce0b34487a4646b717c61f5e9d9e91 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:36.339886 15493 raft_consensus.cc:2454] T a8ce0b34487a4646b717c61f5e9d9e91 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 1.
I20240503 13:56:36.339905 13447 raft_consensus.cc:2454] T a8ce0b34487a4646b717c61f5e9d9e91 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 1.
I20240503 13:56:36.340152 15694 leader_election.cc:302] T a8ce0b34487a4646b717c61f5e9d9e91 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61b887f0d40d497badeca5604c1f0546, 76c386bef61a4bd5a907266a5118ffe2; no voters:
I20240503 13:56:36.340325 18197 raft_consensus.cc:2781] T a8ce0b34487a4646b717c61f5e9d9e91 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:36.340478 18197 raft_consensus.cc:686] T a8ce0b34487a4646b717c61f5e9d9e91 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 LEADER]: Becoming Leader. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Running, Role: LEADER
I20240503 13:56:36.340651 18197 consensus_queue.cc:237] T a8ce0b34487a4646b717c61f5e9d9e91 P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:36.342675 12909 catalog_manager.cc:5484] T a8ce0b34487a4646b717c61f5e9d9e91 P 76c386bef61a4bd5a907266a5118ffe2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71). New cstate: current_term: 1 leader_uuid: "76c386bef61a4bd5a907266a5118ffe2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:36.358146 15604 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 13:56:36.359483 15472 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:56:36.360504 13034 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:36.367538 13424 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:56:36.375692 13295 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:56:36.384161 15868 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 13:56:36.384526 15736 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:56:36.393347 13164 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:36.449638 12909 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:56:36.450784 13051 consensus_queue.cc:237] T fd6a3d908840456f82c346c4bf1d8160 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:36.452067 15493 raft_consensus.cc:1261] T fd6a3d908840456f82c346c4bf1d8160 P 61b887f0d40d497badeca5604c1f0546 [term 2 FOLLOWER]: Refusing update from remote peer 60ba3ffe3b7746469834cd91c84be14e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 13:56:36.452430 13447 raft_consensus.cc:1261] T fd6a3d908840456f82c346c4bf1d8160 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 2 FOLLOWER]: Refusing update from remote peer 60ba3ffe3b7746469834cd91c84be14e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 13:56:36.452431 18545 consensus_queue.cc:1035] T fd6a3d908840456f82c346c4bf1d8160 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:56:36.452848 18545 consensus_queue.cc:1035] T fd6a3d908840456f82c346c4bf1d8160 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:56:36.454605 13447 raft_consensus.cc:1261] T a8ce0b34487a4646b717c61f5e9d9e91 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer 76c386bef61a4bd5a907266a5118ffe2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:36.454787 15493 raft_consensus.cc:1261] T a8ce0b34487a4646b717c61f5e9d9e91 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Refusing update from remote peer 76c386bef61a4bd5a907266a5118ffe2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:36.454844 18134 raft_consensus.cc:2932] T fd6a3d908840456f82c346c4bf1d8160 P 60ba3ffe3b7746469834cd91c84be14e [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:36.455236 13447 raft_consensus.cc:2932] T fd6a3d908840456f82c346c4bf1d8160 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:36.455281 18197 consensus_queue.cc:1035] T a8ce0b34487a4646b717c61f5e9d9e91 P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:36.458220 12908 catalog_manager.cc:5576] TS 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153) does not have the latest schema for tablet a8ce0b34487a4646b717c61f5e9d9e91 (table rebalancer_dynamic_table [id=68ff44ab080b4bbb91c25aa9b06de4d5]). Expected version 1 got 0
I20240503 13:56:36.458212 18193 tablet.cc:1722] T a8ce0b34487a4646b717c61f5e9d9e91 P 76c386bef61a4bd5a907266a5118ffe2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:36.458258 12909 catalog_manager.cc:5484] T fd6a3d908840456f82c346c4bf1d8160 P 7690ee79577c45e3a2c8f8b552c0bff8 reported cstate change: config changed from index 4 to 6, VOTER ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) evicted. New cstate: current_term: 2 leader_uuid: "60ba3ffe3b7746469834cd91c84be14e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:36.458549 18193 tablet.cc:1620] T a8ce0b34487a4646b717c61f5e9d9e91 P 76c386bef61a4bd5a907266a5118ffe2: MemRowSet was empty: no flush needed.
I20240503 13:56:36.458844 18645 consensus_queue.cc:1035] T a8ce0b34487a4646b717c61f5e9d9e91 P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:36.458973 15493 raft_consensus.cc:2932] T fd6a3d908840456f82c346c4bf1d8160 P 61b887f0d40d497badeca5604c1f0546 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:36.459780 18442 tablet.cc:1722] T a8ce0b34487a4646b717c61f5e9d9e91 P 61b887f0d40d497badeca5604c1f0546: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:36.460075 18442 tablet.cc:1620] T a8ce0b34487a4646b717c61f5e9d9e91 P 61b887f0d40d497badeca5604c1f0546: MemRowSet was empty: no flush needed.
I20240503 13:56:36.462469 13295 tablet_service.cc:1514] Processing DeleteTablet for tablet fd6a3d908840456f82c346c4bf1d8160 with delete_type TABLET_DATA_TOMBSTONED (TS ba0194098aa84449ae17e8a0d4c43703 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33250
I20240503 13:56:36.462826 18681 tablet_replica.cc:330] T fd6a3d908840456f82c346c4bf1d8160 P ba0194098aa84449ae17e8a0d4c43703: stopping tablet replica
I20240503 13:56:36.463006 18681 raft_consensus.cc:2229] T fd6a3d908840456f82c346c4bf1d8160 P ba0194098aa84449ae17e8a0d4c43703 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:36.463147 18681 raft_consensus.cc:2258] T fd6a3d908840456f82c346c4bf1d8160 P ba0194098aa84449ae17e8a0d4c43703 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:36.464800 18681 ts_tablet_manager.cc:1875] T fd6a3d908840456f82c346c4bf1d8160 P ba0194098aa84449ae17e8a0d4c43703: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:56:36.465582 12903 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=68ff44ab080b4bbb91c25aa9b06de4d5] alter complete (version 1)
I20240503 13:56:36.467979 18681 ts_tablet_manager.cc:1888] T fd6a3d908840456f82c346c4bf1d8160 P ba0194098aa84449ae17e8a0d4c43703: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 13:56:36.468101 18681 log.cc:1196] T fd6a3d908840456f82c346c4bf1d8160 P ba0194098aa84449ae17e8a0d4c43703: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/wal/wals/fd6a3d908840456f82c346c4bf1d8160
I20240503 13:56:36.468644 12899 catalog_manager.cc:4830] TS ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279): tablet fd6a3d908840456f82c346c4bf1d8160 (table rebalance_test_table_19 [id=e6609c16048348029c01374cb5a68ce8]) successfully deleted
I20240503 13:56:36.469801 12903 catalog_manager.cc:5576] TS 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109) does not have the latest schema for tablet a8ce0b34487a4646b717c61f5e9d9e91 (table rebalancer_dynamic_table [id=68ff44ab080b4bbb91c25aa9b06de4d5]). Expected version 1 got 0
I20240503 13:56:36.471565 18682 tablet.cc:1722] T a8ce0b34487a4646b717c61f5e9d9e91 P 7690ee79577c45e3a2c8f8b552c0bff8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:36.471907 18682 tablet.cc:1620] T a8ce0b34487a4646b717c61f5e9d9e91 P 7690ee79577c45e3a2c8f8b552c0bff8: MemRowSet was empty: no flush needed.
I20240503 13:56:36.560240 18068 consensus_queue.cc:1035] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:36.566869 16404 consensus_queue.cc:1035] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:36.569466 12903 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:56:36.574971 18193 tablet.cc:1722] T a8ce0b34487a4646b717c61f5e9d9e91 P 76c386bef61a4bd5a907266a5118ffe2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:36.575320 18193 tablet.cc:1620] T a8ce0b34487a4646b717c61f5e9d9e91 P 76c386bef61a4bd5a907266a5118ffe2: MemRowSet was empty: no flush needed.
I20240503 13:56:36.577093 18442 tablet.cc:1722] T a8ce0b34487a4646b717c61f5e9d9e91 P 61b887f0d40d497badeca5604c1f0546: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:36.577389 18442 tablet.cc:1620] T a8ce0b34487a4646b717c61f5e9d9e91 P 61b887f0d40d497badeca5604c1f0546: MemRowSet was empty: no flush needed.
I20240503 13:56:36.577620 18682 tablet.cc:1722] T a8ce0b34487a4646b717c61f5e9d9e91 P 7690ee79577c45e3a2c8f8b552c0bff8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:36.577905 18682 tablet.cc:1620] T a8ce0b34487a4646b717c61f5e9d9e91 P 7690ee79577c45e3a2c8f8b552c0bff8: MemRowSet was empty: no flush needed.
I20240503 13:56:36.586971 12903 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=68ff44ab080b4bbb91c25aa9b06de4d5] alter complete (version 2)
I20240503 13:56:36.587908 13034 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:36.595086 15604 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 13:56:36.596364 13424 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:56:36.596879 15472 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:56:36.603010 15736 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:56:36.604468 13295 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:56:36.610631 15868 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 13:56:36.612783 13164 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:36.638357 12906 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_extra_table_20"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:56:36.644953 15868 tablet_service.cc:1467] Processing CreateTablet for tablet a78c6abae54349e8b3de6def2d7aab5f (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=82a12a4e6e70446d89f7f96c193fb3b1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:36.644929 13424 tablet_service.cc:1467] Processing CreateTablet for tablet a78c6abae54349e8b3de6def2d7aab5f (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=82a12a4e6e70446d89f7f96c193fb3b1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:36.645270 15868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a78c6abae54349e8b3de6def2d7aab5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:36.645272 13424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a78c6abae54349e8b3de6def2d7aab5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:36.645915 13032 tablet_service.cc:1467] Processing CreateTablet for tablet d36f17b60df04c2f8965c6cdccfe1204 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=82a12a4e6e70446d89f7f96c193fb3b1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:36.646183 13032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d36f17b60df04c2f8965c6cdccfe1204. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:36.647769 13034 tablet_service.cc:1467] Processing CreateTablet for tablet a78c6abae54349e8b3de6def2d7aab5f (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=82a12a4e6e70446d89f7f96c193fb3b1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:36.648005 13034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a78c6abae54349e8b3de6def2d7aab5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:36.648689 15868 tablet_service.cc:1467] Processing CreateTablet for tablet ce95d734e66b45eaba52b660a456f16e (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=82a12a4e6e70446d89f7f96c193fb3b1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:36.648910 15868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce95d734e66b45eaba52b660a456f16e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:36.649752 15736 tablet_service.cc:1467] Processing CreateTablet for tablet ce95d734e66b45eaba52b660a456f16e (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=82a12a4e6e70446d89f7f96c193fb3b1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:36.649947 18616 tablet_bootstrap.cc:492] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:36.650027 15736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce95d734e66b45eaba52b660a456f16e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:36.650584 18755 tablet_bootstrap.cc:492] T a78c6abae54349e8b3de6def2d7aab5f P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:36.650868 18616 tablet_bootstrap.cc:654] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:36.651309 18755 tablet_bootstrap.cc:654] T a78c6abae54349e8b3de6def2d7aab5f P e47105a0f46e4511b7112e571688a4c7: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:36.652076 18256 tablet_bootstrap.cc:492] T ce95d734e66b45eaba52b660a456f16e P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:36.652119 15604 tablet_service.cc:1467] Processing CreateTablet for tablet d36f17b60df04c2f8965c6cdccfe1204 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=82a12a4e6e70446d89f7f96c193fb3b1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:36.652245 15472 tablet_service.cc:1467] Processing CreateTablet for tablet d36f17b60df04c2f8965c6cdccfe1204 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=82a12a4e6e70446d89f7f96c193fb3b1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:36.652364 15604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d36f17b60df04c2f8965c6cdccfe1204. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:36.652541 15472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d36f17b60df04c2f8965c6cdccfe1204. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:36.652662 18756 tablet_bootstrap.cc:492] T d36f17b60df04c2f8965c6cdccfe1204 P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:36.652947 18256 tablet_bootstrap.cc:654] T ce95d734e66b45eaba52b660a456f16e P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:36.653643 18756 tablet_bootstrap.cc:654] T d36f17b60df04c2f8965c6cdccfe1204 P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:36.654832 18615 tablet_bootstrap.cc:492] T d36f17b60df04c2f8965c6cdccfe1204 P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:36.655130 18756 tablet_bootstrap.cc:492] T d36f17b60df04c2f8965c6cdccfe1204 P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:36.655217 18756 ts_tablet_manager.cc:1367] T d36f17b60df04c2f8965c6cdccfe1204 P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 13:56:36.655359 18616 tablet_bootstrap.cc:492] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:36.655447 18616 ts_tablet_manager.cc:1367] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 13:56:36.655476 15605 tablet_service.cc:1467] Processing CreateTablet for tablet ce95d734e66b45eaba52b660a456f16e (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=82a12a4e6e70446d89f7f96c193fb3b1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:36.655721 15605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce95d734e66b45eaba52b660a456f16e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:36.655669 18756 raft_consensus.cc:348] T d36f17b60df04c2f8965c6cdccfe1204 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:36.655766 18615 tablet_bootstrap.cc:654] T d36f17b60df04c2f8965c6cdccfe1204 P 61b887f0d40d497badeca5604c1f0546: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:36.655771 18756 raft_consensus.cc:374] T d36f17b60df04c2f8965c6cdccfe1204 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:36.655802 18756 raft_consensus.cc:729] T d36f17b60df04c2f8965c6cdccfe1204 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:36.655964 18756 consensus_queue.cc:260] T d36f17b60df04c2f8965c6cdccfe1204 P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:36.655956 18616 raft_consensus.cc:348] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:36.656059 18616 raft_consensus.cc:374] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:36.656087 18616 raft_consensus.cc:729] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:36.656317 18756 ts_tablet_manager.cc:1398] T d36f17b60df04c2f8965c6cdccfe1204 P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:36.656459 18756 tablet_bootstrap.cc:492] T a78c6abae54349e8b3de6def2d7aab5f P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:36.656792 18615 tablet_bootstrap.cc:492] T d36f17b60df04c2f8965c6cdccfe1204 P 61b887f0d40d497badeca5604c1f0546: No bootstrap required, opened a new log
I20240503 13:56:36.656867 18615 ts_tablet_manager.cc:1367] T d36f17b60df04c2f8965c6cdccfe1204 P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:36.657320 18615 raft_consensus.cc:348] T d36f17b60df04c2f8965c6cdccfe1204 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:36.657418 18615 raft_consensus.cc:374] T d36f17b60df04c2f8965c6cdccfe1204 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:36.657444 18615 raft_consensus.cc:729] T d36f17b60df04c2f8965c6cdccfe1204 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:36.657541 18615 consensus_queue.cc:260] T d36f17b60df04c2f8965c6cdccfe1204 P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:36.657805 18615 ts_tablet_manager.cc:1398] T d36f17b60df04c2f8965c6cdccfe1204 P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:36.658336 18756 tablet_bootstrap.cc:654] T a78c6abae54349e8b3de6def2d7aab5f P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:36.659122 18761 tablet_bootstrap.cc:492] T d36f17b60df04c2f8965c6cdccfe1204 P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:36.660063 18761 tablet_bootstrap.cc:654] T d36f17b60df04c2f8965c6cdccfe1204 P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:36.659991 18616 consensus_queue.cc:260] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:36.661074 18616 ts_tablet_manager.cc:1398] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240503 13:56:36.661418 18761 tablet_bootstrap.cc:492] T d36f17b60df04c2f8965c6cdccfe1204 P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:36.661559 18761 ts_tablet_manager.cc:1367] T d36f17b60df04c2f8965c6cdccfe1204 P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:56:36.661947 18755 tablet_bootstrap.cc:492] T a78c6abae54349e8b3de6def2d7aab5f P e47105a0f46e4511b7112e571688a4c7: No bootstrap required, opened a new log
I20240503 13:56:36.662030 18761 raft_consensus.cc:348] T d36f17b60df04c2f8965c6cdccfe1204 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:36.662324 18761 raft_consensus.cc:374] T d36f17b60df04c2f8965c6cdccfe1204 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:36.662405 18761 raft_consensus.cc:729] T d36f17b60df04c2f8965c6cdccfe1204 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:36.662576 18761 consensus_queue.cc:260] T d36f17b60df04c2f8965c6cdccfe1204 P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:36.662103 18755 ts_tablet_manager.cc:1367] T a78c6abae54349e8b3de6def2d7aab5f P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 13:56:36.664165 18755 raft_consensus.cc:348] T a78c6abae54349e8b3de6def2d7aab5f P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:36.664355 18755 raft_consensus.cc:374] T a78c6abae54349e8b3de6def2d7aab5f P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:36.664448 18755 raft_consensus.cc:729] T a78c6abae54349e8b3de6def2d7aab5f P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:36.664806 18755 consensus_queue.cc:260] T a78c6abae54349e8b3de6def2d7aab5f P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:36.665549 18755 ts_tablet_manager.cc:1398] T a78c6abae54349e8b3de6def2d7aab5f P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:36.665680 18256 tablet_bootstrap.cc:492] T ce95d734e66b45eaba52b660a456f16e P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:36.665796 18256 ts_tablet_manager.cc:1367] T ce95d734e66b45eaba52b660a456f16e P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240503 13:56:36.665796 18755 tablet_bootstrap.cc:492] T ce95d734e66b45eaba52b660a456f16e P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:36.666215 18256 raft_consensus.cc:348] T ce95d734e66b45eaba52b660a456f16e P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:36.666311 18256 raft_consensus.cc:374] T ce95d734e66b45eaba52b660a456f16e P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:36.666335 18256 raft_consensus.cc:729] T ce95d734e66b45eaba52b660a456f16e P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:36.666452 18256 consensus_queue.cc:260] T ce95d734e66b45eaba52b660a456f16e P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:36.666723 18256 ts_tablet_manager.cc:1398] T ce95d734e66b45eaba52b660a456f16e P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:36.667086 18755 tablet_bootstrap.cc:654] T ce95d734e66b45eaba52b660a456f16e P e47105a0f46e4511b7112e571688a4c7: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:36.668874 18755 tablet_bootstrap.cc:492] T ce95d734e66b45eaba52b660a456f16e P e47105a0f46e4511b7112e571688a4c7: No bootstrap required, opened a new log
I20240503 13:56:36.669029 18755 ts_tablet_manager.cc:1367] T ce95d734e66b45eaba52b660a456f16e P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20240503 13:56:36.669294 18756 tablet_bootstrap.cc:492] T a78c6abae54349e8b3de6def2d7aab5f P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:36.669445 18756 ts_tablet_manager.cc:1367] T a78c6abae54349e8b3de6def2d7aab5f P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 13:56:36.669914 18756 raft_consensus.cc:348] T a78c6abae54349e8b3de6def2d7aab5f P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:36.670022 18756 raft_consensus.cc:374] T a78c6abae54349e8b3de6def2d7aab5f P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:36.670048 18756 raft_consensus.cc:729] T a78c6abae54349e8b3de6def2d7aab5f P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:36.670156 18756 consensus_queue.cc:260] T a78c6abae54349e8b3de6def2d7aab5f P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:36.670406 18756 ts_tablet_manager.cc:1398] T a78c6abae54349e8b3de6def2d7aab5f P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:36.670517 16404 consensus_queue.cc:1035] T 10ee7dee14364389871900670a02906a P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:36.671900 18761 ts_tablet_manager.cc:1398] T d36f17b60df04c2f8965c6cdccfe1204 P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20240503 13:56:36.669780 18755 raft_consensus.cc:348] T ce95d734e66b45eaba52b660a456f16e P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:36.672637 18755 raft_consensus.cc:374] T ce95d734e66b45eaba52b660a456f16e P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:36.672722 18755 raft_consensus.cc:729] T ce95d734e66b45eaba52b660a456f16e P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:36.673723 18761 tablet_bootstrap.cc:492] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:36.674837 18761 tablet_bootstrap.cc:654] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:36.675635 18068 consensus_queue.cc:1035] T 10ee7dee14364389871900670a02906a P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:36.676244 18761 tablet_bootstrap.cc:492] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:36.676329 18761 ts_tablet_manager.cc:1367] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:56:36.676803 18761 raft_consensus.cc:348] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:36.676916 18761 raft_consensus.cc:374] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:36.676946 18761 raft_consensus.cc:729] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:36.677068 18761 consensus_queue.cc:260] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:36.677453 18761 ts_tablet_manager.cc:1398] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:36.677760 18755 consensus_queue.cc:260] T ce95d734e66b45eaba52b660a456f16e P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:36.678145 18755 ts_tablet_manager.cc:1398] T ce95d734e66b45eaba52b660a456f16e P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.009s user 0.000s sys 0.001s
I20240503 13:56:36.689095 12907 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:56:36.695464 18193 tablet.cc:1722] T a8ce0b34487a4646b717c61f5e9d9e91 P 76c386bef61a4bd5a907266a5118ffe2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:36.695834 18442 tablet.cc:1722] T a8ce0b34487a4646b717c61f5e9d9e91 P 61b887f0d40d497badeca5604c1f0546: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:36.696676 18682 tablet.cc:1722] T a8ce0b34487a4646b717c61f5e9d9e91 P 7690ee79577c45e3a2c8f8b552c0bff8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:36.702533 12907 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table_new_name [id=68ff44ab080b4bbb91c25aa9b06de4d5] alter complete (version 3)
I20240503 13:56:36.702699 15493 consensus_queue.cc:237] T fdc5f76f63ec4528ad57202555caf243 P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:36.704314 13051 raft_consensus.cc:1261] T fdc5f76f63ec4528ad57202555caf243 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Refusing update from remote peer 61b887f0d40d497badeca5604c1f0546: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:56:36.704427 13183 raft_consensus.cc:1261] T fdc5f76f63ec4528ad57202555caf243 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Refusing update from remote peer 61b887f0d40d497badeca5604c1f0546: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:56:36.704811 18789 consensus_queue.cc:1035] T fdc5f76f63ec4528ad57202555caf243 P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:56:36.705214 18789 consensus_queue.cc:1035] T fdc5f76f63ec4528ad57202555caf243 P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:56:36.705459 18771 consensus_queue.cc:1035] T af1af9a911c343e880441f2177383b25 P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:36.707309 18789 raft_consensus.cc:2932] T fdc5f76f63ec4528ad57202555caf243 P 61b887f0d40d497badeca5604c1f0546 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } }
I20240503 13:56:36.708670 13051 raft_consensus.cc:2932] T fdc5f76f63ec4528ad57202555caf243 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } }
I20240503 13:56:36.710249 18771 consensus_queue.cc:1035] T af1af9a911c343e880441f2177383b25 P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:36.711992 12903 catalog_manager.cc:5484] T fdc5f76f63ec4528ad57202555caf243 P 60ba3ffe3b7746469834cd91c84be14e reported cstate change: config changed from index 2 to 3, VOTER 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71) evicted. New cstate: current_term: 1 leader_uuid: "61b887f0d40d497badeca5604c1f0546" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } }
I20240503 13:56:36.714866 13183 raft_consensus.cc:2932] T fdc5f76f63ec4528ad57202555caf243 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } }
I20240503 13:56:36.716341 15736 tablet_service.cc:1514] Processing DeleteTablet for tablet fdc5f76f63ec4528ad57202555caf243 with delete_type TABLET_DATA_TOMBSTONED (TS 76c386bef61a4bd5a907266a5118ffe2 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:44984
I20240503 13:56:36.717151 18799 tablet_replica.cc:330] T fdc5f76f63ec4528ad57202555caf243 P 76c386bef61a4bd5a907266a5118ffe2: stopping tablet replica
I20240503 13:56:36.717298 18799 raft_consensus.cc:2229] T fdc5f76f63ec4528ad57202555caf243 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:36.717454 18799 raft_consensus.cc:2258] T fdc5f76f63ec4528ad57202555caf243 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:36.717891 18799 ts_tablet_manager.cc:1875] T fdc5f76f63ec4528ad57202555caf243 P 76c386bef61a4bd5a907266a5118ffe2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:56:36.720014 18799 ts_tablet_manager.cc:1888] T fdc5f76f63ec4528ad57202555caf243 P 76c386bef61a4bd5a907266a5118ffe2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 13:56:36.720511 18799 log.cc:1196] T fdc5f76f63ec4528ad57202555caf243 P 76c386bef61a4bd5a907266a5118ffe2: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/wal/wals/fdc5f76f63ec4528ad57202555caf243
I20240503 13:56:36.721030 12899 catalog_manager.cc:4830] TS 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885): tablet fdc5f76f63ec4528ad57202555caf243 (table rebalancer_extra_table_6 [id=1161fd279f6d435887536e9c124d251a]) successfully deleted
I20240503 13:56:36.785921 18771 raft_consensus.cc:482] T d36f17b60df04c2f8965c6cdccfe1204 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:36.786059 18771 raft_consensus.cc:504] T d36f17b60df04c2f8965c6cdccfe1204 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:36.786490 18771 leader_election.cc:288] T d36f17b60df04c2f8965c6cdccfe1204 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153), 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:36.787765 13051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d36f17b60df04c2f8965c6cdccfe1204" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e" is_pre_election: true
I20240503 13:56:36.787917 13051 raft_consensus.cc:2454] T d36f17b60df04c2f8965c6cdccfe1204 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 0.
I20240503 13:56:36.793207 15559 leader_election.cc:302] T d36f17b60df04c2f8965c6cdccfe1204 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 60ba3ffe3b7746469834cd91c84be14e; no voters:
I20240503 13:56:36.793444 18771 raft_consensus.cc:2781] T d36f17b60df04c2f8965c6cdccfe1204 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:36.793524 18771 raft_consensus.cc:482] T d36f17b60df04c2f8965c6cdccfe1204 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:36.793556 18771 raft_consensus.cc:3037] T d36f17b60df04c2f8965c6cdccfe1204 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:36.794776 18771 raft_consensus.cc:504] T d36f17b60df04c2f8965c6cdccfe1204 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:36.795011 15493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d36f17b60df04c2f8965c6cdccfe1204" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546" is_pre_election: true
I20240503 13:56:36.795145 15493 raft_consensus.cc:2454] T d36f17b60df04c2f8965c6cdccfe1204 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 0.
I20240503 13:56:36.795230 18771 leader_election.cc:288] T d36f17b60df04c2f8965c6cdccfe1204 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 election: Requested vote from peers 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153), 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:36.795739 13051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d36f17b60df04c2f8965c6cdccfe1204" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e"
I20240503 13:56:36.795864 13051 raft_consensus.cc:3037] T d36f17b60df04c2f8965c6cdccfe1204 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:36.796916 13051 raft_consensus.cc:2454] T d36f17b60df04c2f8965c6cdccfe1204 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 1.
I20240503 13:56:36.797433 15559 leader_election.cc:302] T d36f17b60df04c2f8965c6cdccfe1204 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 60ba3ffe3b7746469834cd91c84be14e; no voters:
I20240503 13:56:36.797477 15493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d36f17b60df04c2f8965c6cdccfe1204" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546"
I20240503 13:56:36.797609 15493 raft_consensus.cc:3037] T d36f17b60df04c2f8965c6cdccfe1204 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:36.797621 18771 raft_consensus.cc:2781] T d36f17b60df04c2f8965c6cdccfe1204 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:36.797685 18771 raft_consensus.cc:686] T d36f17b60df04c2f8965c6cdccfe1204 P 009f849ee7b54c80a3f5264695047e4d [term 1 LEADER]: Becoming Leader. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Running, Role: LEADER
I20240503 13:56:36.797817 18771 consensus_queue.cc:237] T d36f17b60df04c2f8965c6cdccfe1204 P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:36.798651 15493 raft_consensus.cc:2454] T d36f17b60df04c2f8965c6cdccfe1204 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 1.
I20240503 13:56:36.800338 12904 catalog_manager.cc:5484] T d36f17b60df04c2f8965c6cdccfe1204 P 009f849ee7b54c80a3f5264695047e4d reported cstate change: term changed from 0 to 1, leader changed from <none> to 009f849ee7b54c80a3f5264695047e4d (127.6.98.70). New cstate: current_term: 1 leader_uuid: "009f849ee7b54c80a3f5264695047e4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } }
I20240503 13:56:36.805593 12904 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:36.805750 12904 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:36.809999 12904 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Sending DeleteTablet for 3 replicas of tablet a8ce0b34487a4646b717c61f5e9d9e91
I20240503 13:56:36.810604 15472 tablet_service.cc:1514] Processing DeleteTablet for tablet a8ce0b34487a4646b717c61f5e9d9e91 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:36 UTC) from {username='slave'} at 127.0.0.1:58758
I20240503 13:56:36.810717 13424 tablet_service.cc:1514] Processing DeleteTablet for tablet a8ce0b34487a4646b717c61f5e9d9e91 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:36 UTC) from {username='slave'} at 127.0.0.1:33678
I20240503 13:56:36.811450 18854 tablet_replica.cc:330] T a8ce0b34487a4646b717c61f5e9d9e91 P 7690ee79577c45e3a2c8f8b552c0bff8: stopping tablet replica
I20240503 13:56:36.811507 15736 tablet_service.cc:1514] Processing DeleteTablet for tablet a8ce0b34487a4646b717c61f5e9d9e91 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:36 UTC) from {username='slave'} at 127.0.0.1:44984
I20240503 13:56:36.811607 18854 raft_consensus.cc:2229] T a8ce0b34487a4646b717c61f5e9d9e91 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:36.811705 18799 tablet_replica.cc:330] T a8ce0b34487a4646b717c61f5e9d9e91 P 76c386bef61a4bd5a907266a5118ffe2: stopping tablet replica
I20240503 13:56:36.811735 18854 raft_consensus.cc:2258] T a8ce0b34487a4646b717c61f5e9d9e91 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:36.811827 18799 raft_consensus.cc:2229] T a8ce0b34487a4646b717c61f5e9d9e91 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:56:36.812060 18799 raft_consensus.cc:2258] T a8ce0b34487a4646b717c61f5e9d9e91 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:36.812700 18799 ts_tablet_manager.cc:1875] T a8ce0b34487a4646b717c61f5e9d9e91 P 76c386bef61a4bd5a907266a5118ffe2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:36.814769 18799 ts_tablet_manager.cc:1888] T a8ce0b34487a4646b717c61f5e9d9e91 P 76c386bef61a4bd5a907266a5118ffe2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:36.814841 18799 log.cc:1196] T a8ce0b34487a4646b717c61f5e9d9e91 P 76c386bef61a4bd5a907266a5118ffe2: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/wal/wals/a8ce0b34487a4646b717c61f5e9d9e91
I20240503 13:56:36.815091 18799 ts_tablet_manager.cc:1909] T a8ce0b34487a4646b717c61f5e9d9e91 P 76c386bef61a4bd5a907266a5118ffe2: Deleting consensus metadata
I20240503 13:56:36.816308 12899 catalog_manager.cc:4830] TS 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885): tablet a8ce0b34487a4646b717c61f5e9d9e91 (table rebalancer_dynamic_table_new_name [id=68ff44ab080b4bbb91c25aa9b06de4d5]) successfully deleted
I20240503 13:56:36.818253 18854 ts_tablet_manager.cc:1875] T a8ce0b34487a4646b717c61f5e9d9e91 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:36.818509 18374 raft_consensus.cc:482] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:36.818586 18374 raft_consensus.cc:504] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:36.819041 18374 leader_election.cc:288] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:36.819694 13051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a78c6abae54349e8b3de6def2d7aab5f" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e" is_pre_election: true
I20240503 13:56:36.819831 13051 raft_consensus.cc:2454] T a78c6abae54349e8b3de6def2d7aab5f P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 0.
I20240503 13:56:36.820134 13381 leader_election.cc:302] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 7690ee79577c45e3a2c8f8b552c0bff8; no voters:
I20240503 13:56:36.820329 18374 raft_consensus.cc:2781] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:36.820379 18374 raft_consensus.cc:482] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:36.820415 18374 raft_consensus.cc:3037] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:36.820576 18854 ts_tablet_manager.cc:1888] T a8ce0b34487a4646b717c61f5e9d9e91 P 7690ee79577c45e3a2c8f8b552c0bff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:36.820638 18854 log.cc:1196] T a8ce0b34487a4646b717c61f5e9d9e91 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal/wals/a8ce0b34487a4646b717c61f5e9d9e91
I20240503 13:56:36.820950 18854 ts_tablet_manager.cc:1909] T a8ce0b34487a4646b717c61f5e9d9e91 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting consensus metadata
I20240503 13:56:36.821169 15889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a78c6abae54349e8b3de6def2d7aab5f" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7" is_pre_election: true
I20240503 13:56:36.821223 18853 tablet_replica.cc:330] T a8ce0b34487a4646b717c61f5e9d9e91 P 61b887f0d40d497badeca5604c1f0546: stopping tablet replica
I20240503 13:56:36.821298 15889 raft_consensus.cc:2454] T a78c6abae54349e8b3de6def2d7aab5f P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 0.
I20240503 13:56:36.821332 18374 raft_consensus.cc:504] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:36.821411 18853 raft_consensus.cc:2229] T a8ce0b34487a4646b717c61f5e9d9e91 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:36.821580 18853 raft_consensus.cc:2258] T a8ce0b34487a4646b717c61f5e9d9e91 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:36.821691 18374 leader_election.cc:288] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 election: Requested vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:36.822736 18853 ts_tablet_manager.cc:1875] T a8ce0b34487a4646b717c61f5e9d9e91 P 61b887f0d40d497badeca5604c1f0546: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:36.822880 12896 catalog_manager.cc:4830] TS 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109): tablet a8ce0b34487a4646b717c61f5e9d9e91 (table rebalancer_dynamic_table_new_name [id=68ff44ab080b4bbb91c25aa9b06de4d5]) successfully deleted
I20240503 13:56:36.823274 15889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a78c6abae54349e8b3de6def2d7aab5f" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7"
I20240503 13:56:36.823382 15889 raft_consensus.cc:3037] T a78c6abae54349e8b3de6def2d7aab5f P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:36.824430 15889 raft_consensus.cc:2454] T a78c6abae54349e8b3de6def2d7aab5f P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 1.
I20240503 13:56:36.824414 13051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a78c6abae54349e8b3de6def2d7aab5f" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e"
I20240503 13:56:36.824553 13051 raft_consensus.cc:3037] T a78c6abae54349e8b3de6def2d7aab5f P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:36.825054 13380 leader_election.cc:302] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:36.825273 18374 raft_consensus.cc:2781] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:36.825423 18374 raft_consensus.cc:686] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 LEADER]: Becoming Leader. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Running, Role: LEADER
I20240503 13:56:36.825565 18374 consensus_queue.cc:237] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:36.825619 13051 raft_consensus.cc:2454] T a78c6abae54349e8b3de6def2d7aab5f P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 1.
I20240503 13:56:36.827708 12908 catalog_manager.cc:5484] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68). New cstate: current_term: 1 leader_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } }
I20240503 13:56:36.827872 18853 ts_tablet_manager.cc:1888] T a8ce0b34487a4646b717c61f5e9d9e91 P 61b887f0d40d497badeca5604c1f0546: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:36.827936 18853 log.cc:1196] T a8ce0b34487a4646b717c61f5e9d9e91 P 61b887f0d40d497badeca5604c1f0546: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/wal/wals/a8ce0b34487a4646b717c61f5e9d9e91
I20240503 13:56:36.828198 18853 ts_tablet_manager.cc:1909] T a8ce0b34487a4646b717c61f5e9d9e91 P 61b887f0d40d497badeca5604c1f0546: Deleting consensus metadata
I20240503 13:56:36.835196 12899 catalog_manager.cc:4830] TS 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153): tablet a8ce0b34487a4646b717c61f5e9d9e91 (table rebalancer_dynamic_table_new_name [id=68ff44ab080b4bbb91c25aa9b06de4d5]) successfully deleted
I20240503 13:56:36.884217 18771 raft_consensus.cc:482] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:36.884346 18771 raft_consensus.cc:504] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:36.884980 18771 leader_election.cc:288] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:36.885380 15889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce95d734e66b45eaba52b660a456f16e" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7" is_pre_election: true
I20240503 13:56:36.885380 15758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce95d734e66b45eaba52b660a456f16e" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2" is_pre_election: true
I20240503 13:56:36.885529 15758 raft_consensus.cc:2454] T ce95d734e66b45eaba52b660a456f16e P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 0.
I20240503 13:56:36.885538 15889 raft_consensus.cc:2454] T ce95d734e66b45eaba52b660a456f16e P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 0.
I20240503 13:56:36.885807 15558 leader_election.cc:302] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:36.885978 18771 raft_consensus.cc:2781] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:36.886058 18771 raft_consensus.cc:482] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:36.886085 18771 raft_consensus.cc:3037] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:36.886965 18771 raft_consensus.cc:504] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:36.887286 18771 leader_election.cc:288] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 election: Requested vote from peers 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:36.887652 15889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce95d734e66b45eaba52b660a456f16e" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7"
I20240503 13:56:36.887697 15758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce95d734e66b45eaba52b660a456f16e" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2"
I20240503 13:56:36.887771 15889 raft_consensus.cc:3037] T ce95d734e66b45eaba52b660a456f16e P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:36.887826 15758 raft_consensus.cc:3037] T ce95d734e66b45eaba52b660a456f16e P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:36.888566 15889 raft_consensus.cc:2454] T ce95d734e66b45eaba52b660a456f16e P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 1.
I20240503 13:56:36.888775 15758 raft_consensus.cc:2454] T ce95d734e66b45eaba52b660a456f16e P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 1.
I20240503 13:56:36.888813 15558 leader_election.cc:302] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:36.888947 18771 raft_consensus.cc:2781] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:36.889042 18771 raft_consensus.cc:686] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d [term 1 LEADER]: Becoming Leader. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Running, Role: LEADER
I20240503 13:56:36.889183 18771 consensus_queue.cc:237] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:36.891237 12912 catalog_manager.cc:5484] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d reported cstate change: term changed from 0 to 1, leader changed from <none> to 009f849ee7b54c80a3f5264695047e4d (127.6.98.70). New cstate: current_term: 1 leader_uuid: "009f849ee7b54c80a3f5264695047e4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:36.912667 12912 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:56:36.919612 15605 tablet_service.cc:1467] Processing CreateTablet for tablet dbcdf769be2b4fe1b4b97f5c98673594 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2c10732c24b04c59b89993df2621459d]), partition=
I20240503 13:56:36.919894 15605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbcdf769be2b4fe1b4b97f5c98673594. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:36.920953 15868 tablet_service.cc:1467] Processing CreateTablet for tablet dbcdf769be2b4fe1b4b97f5c98673594 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2c10732c24b04c59b89993df2621459d]), partition=
I20240503 13:56:36.921026 15472 tablet_service.cc:1467] Processing CreateTablet for tablet dbcdf769be2b4fe1b4b97f5c98673594 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2c10732c24b04c59b89993df2621459d]), partition=
I20240503 13:56:36.921252 15472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbcdf769be2b4fe1b4b97f5c98673594. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:36.921254 15868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbcdf769be2b4fe1b4b97f5c98673594. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:36.923339 18761 tablet_bootstrap.cc:492] T dbcdf769be2b4fe1b4b97f5c98673594 P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:36.923389 18615 tablet_bootstrap.cc:492] T dbcdf769be2b4fe1b4b97f5c98673594 P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:36.924280 18761 tablet_bootstrap.cc:654] T dbcdf769be2b4fe1b4b97f5c98673594 P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:36.924331 18615 tablet_bootstrap.cc:654] T dbcdf769be2b4fe1b4b97f5c98673594 P 61b887f0d40d497badeca5604c1f0546: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:36.924453 18755 tablet_bootstrap.cc:492] T dbcdf769be2b4fe1b4b97f5c98673594 P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:36.925380 18755 tablet_bootstrap.cc:654] T dbcdf769be2b4fe1b4b97f5c98673594 P e47105a0f46e4511b7112e571688a4c7: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:36.926594 13424 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 13:56:36.930708 18615 tablet_bootstrap.cc:492] T dbcdf769be2b4fe1b4b97f5c98673594 P 61b887f0d40d497badeca5604c1f0546: No bootstrap required, opened a new log
I20240503 13:56:36.930802 18615 ts_tablet_manager.cc:1367] T dbcdf769be2b4fe1b4b97f5c98673594 P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20240503 13:56:36.931216 18615 raft_consensus.cc:348] T dbcdf769be2b4fe1b4b97f5c98673594 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:36.931342 18615 raft_consensus.cc:374] T dbcdf769be2b4fe1b4b97f5c98673594 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:36.931368 18615 raft_consensus.cc:729] T dbcdf769be2b4fe1b4b97f5c98673594 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:36.931474 18615 consensus_queue.cc:260] T dbcdf769be2b4fe1b4b97f5c98673594 P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:36.931728 18615 ts_tablet_manager.cc:1398] T dbcdf769be2b4fe1b4b97f5c98673594 P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:36.932104 15736 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:56:36.935461 15472 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:56:36.935775 13164 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:36.939469 18761 tablet_bootstrap.cc:492] T dbcdf769be2b4fe1b4b97f5c98673594 P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:36.939589 18761 ts_tablet_manager.cc:1367] T dbcdf769be2b4fe1b4b97f5c98673594 P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.002s
I20240503 13:56:36.940110 18761 raft_consensus.cc:348] T dbcdf769be2b4fe1b4b97f5c98673594 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:36.940232 18761 raft_consensus.cc:374] T dbcdf769be2b4fe1b4b97f5c98673594 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:36.940263 18761 raft_consensus.cc:729] T dbcdf769be2b4fe1b4b97f5c98673594 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:36.940378 18761 consensus_queue.cc:260] T dbcdf769be2b4fe1b4b97f5c98673594 P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:36.940706 18761 ts_tablet_manager.cc:1398] T dbcdf769be2b4fe1b4b97f5c98673594 P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:36.944031 18755 tablet_bootstrap.cc:492] T dbcdf769be2b4fe1b4b97f5c98673594 P e47105a0f46e4511b7112e571688a4c7: No bootstrap required, opened a new log
I20240503 13:56:36.944131 18755 ts_tablet_manager.cc:1367] T dbcdf769be2b4fe1b4b97f5c98673594 P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20240503 13:56:36.944653 18755 raft_consensus.cc:348] T dbcdf769be2b4fe1b4b97f5c98673594 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:36.944762 18755 raft_consensus.cc:374] T dbcdf769be2b4fe1b4b97f5c98673594 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:36.944789 18755 raft_consensus.cc:729] T dbcdf769be2b4fe1b4b97f5c98673594 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:36.944892 18755 consensus_queue.cc:260] T dbcdf769be2b4fe1b4b97f5c98673594 P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:36.947810 18755 ts_tablet_manager.cc:1398] T dbcdf769be2b4fe1b4b97f5c98673594 P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 13:56:36.949393 13034 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:36.950366 15868 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 13:56:36.954712 15605 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:36.959297 13295 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:56:37.021221 15493 consensus_queue.cc:237] T 9eb0d911701b4b2bb33a6b7ab59f08aa P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:37.022553 13183 raft_consensus.cc:1261] T 9eb0d911701b4b2bb33a6b7ab59f08aa P 8ccf5a4ac450414c9d3579bd5800b770 [term 2 FOLLOWER]: Refusing update from remote peer 61b887f0d40d497badeca5604c1f0546: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 13:56:37.022653 13447 raft_consensus.cc:1261] T 9eb0d911701b4b2bb33a6b7ab59f08aa P 7690ee79577c45e3a2c8f8b552c0bff8 [term 2 FOLLOWER]: Refusing update from remote peer 61b887f0d40d497badeca5604c1f0546: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 13:56:37.022969 17385 consensus_queue.cc:1035] T 9eb0d911701b4b2bb33a6b7ab59f08aa P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:56:37.023275 18789 consensus_queue.cc:1035] T 9eb0d911701b4b2bb33a6b7ab59f08aa P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:56:37.025223 18787 raft_consensus.cc:2932] T 9eb0d911701b4b2bb33a6b7ab59f08aa P 61b887f0d40d497badeca5604c1f0546 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:37.025621 13183 raft_consensus.cc:2932] T 9eb0d911701b4b2bb33a6b7ab59f08aa P 8ccf5a4ac450414c9d3579bd5800b770 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:37.026736 13447 raft_consensus.cc:2932] T 9eb0d911701b4b2bb33a6b7ab59f08aa P 7690ee79577c45e3a2c8f8b552c0bff8 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:37.027838 12912 catalog_manager.cc:5484] T 9eb0d911701b4b2bb33a6b7ab59f08aa P 61b887f0d40d497badeca5604c1f0546 reported cstate change: config changed from index 4 to 6, VOTER ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) evicted. New cstate: current_term: 2 leader_uuid: "61b887f0d40d497badeca5604c1f0546" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:37.030707 13295 tablet_service.cc:1514] Processing DeleteTablet for tablet 9eb0d911701b4b2bb33a6b7ab59f08aa with delete_type TABLET_DATA_TOMBSTONED (TS ba0194098aa84449ae17e8a0d4c43703 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33250
I20240503 13:56:37.031114 18892 tablet_replica.cc:330] T 9eb0d911701b4b2bb33a6b7ab59f08aa P ba0194098aa84449ae17e8a0d4c43703: stopping tablet replica
I20240503 13:56:37.031246 18892 raft_consensus.cc:2229] T 9eb0d911701b4b2bb33a6b7ab59f08aa P ba0194098aa84449ae17e8a0d4c43703 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:37.031352 18892 raft_consensus.cc:2258] T 9eb0d911701b4b2bb33a6b7ab59f08aa P ba0194098aa84449ae17e8a0d4c43703 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:37.031839 18892 ts_tablet_manager.cc:1875] T 9eb0d911701b4b2bb33a6b7ab59f08aa P ba0194098aa84449ae17e8a0d4c43703: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:56:37.033915 18892 ts_tablet_manager.cc:1888] T 9eb0d911701b4b2bb33a6b7ab59f08aa P ba0194098aa84449ae17e8a0d4c43703: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 13:56:37.034046 18892 log.cc:1196] T 9eb0d911701b4b2bb33a6b7ab59f08aa P ba0194098aa84449ae17e8a0d4c43703: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/wal/wals/9eb0d911701b4b2bb33a6b7ab59f08aa
I20240503 13:56:37.034646 12899 catalog_manager.cc:4830] TS ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279): tablet 9eb0d911701b4b2bb33a6b7ab59f08aa (table rebalance_test_table_5 [id=ef32d41e01cb46ed894c4ff29e5c0e52]) successfully deleted
I20240503 13:56:37.059212 13314 consensus_queue.cc:237] T dc544319be0b484f8e9e5a3fbc40f530 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:37.061069 13051 raft_consensus.cc:1261] T dc544319be0b484f8e9e5a3fbc40f530 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:56:37.061240 13183 raft_consensus.cc:1261] T dc544319be0b484f8e9e5a3fbc40f530 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 13:56:37.061323 13252 consensus_peers.cc:485] T dc544319be0b484f8e9e5a3fbc40f530 P ba0194098aa84449ae17e8a0d4c43703 -> Peer 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885): Couldn't send request to peer 76c386bef61a4bd5a907266a5118ffe2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dc544319be0b484f8e9e5a3fbc40f530. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:37.061501 18592 consensus_queue.cc:1035] T dc544319be0b484f8e9e5a3fbc40f530 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:37.062075 18592 consensus_queue.cc:1035] T dc544319be0b484f8e9e5a3fbc40f530 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:37.064034 18873 raft_consensus.cc:2932] T dc544319be0b484f8e9e5a3fbc40f530 P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:37.064627 13183 raft_consensus.cc:2932] T dc544319be0b484f8e9e5a3fbc40f530 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:37.067381 13051 raft_consensus.cc:2932] T dc544319be0b484f8e9e5a3fbc40f530 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:37.067672 12906 catalog_manager.cc:5484] T dc544319be0b484f8e9e5a3fbc40f530 P 8ccf5a4ac450414c9d3579bd5800b770 reported cstate change: config changed from index -1 to 4, VOTER 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71) added. New cstate: current_term: 1 leader_uuid: "ba0194098aa84449ae17e8a0d4c43703" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:37.088698 15758 consensus_queue.cc:237] T 0878ccb6135f4485b5ba35ae67a76d0c P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:37.090183 13051 raft_consensus.cc:1261] T 0878ccb6135f4485b5ba35ae67a76d0c P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Refusing update from remote peer 76c386bef61a4bd5a907266a5118ffe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:37.090171 13447 raft_consensus.cc:1261] T 0878ccb6135f4485b5ba35ae67a76d0c P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer 76c386bef61a4bd5a907266a5118ffe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 13:56:37.090526 15694 consensus_peers.cc:485] T 0878ccb6135f4485b5ba35ae67a76d0c P 76c386bef61a4bd5a907266a5118ffe2 -> Peer ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279): Couldn't send request to peer ba0194098aa84449ae17e8a0d4c43703. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0878ccb6135f4485b5ba35ae67a76d0c. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:37.090590 18645 consensus_queue.cc:1035] T 0878ccb6135f4485b5ba35ae67a76d0c P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:56:37.090718 18068 raft_consensus.cc:482] T dbcdf769be2b4fe1b4b97f5c98673594 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:37.090806 18068 raft_consensus.cc:504] T dbcdf769be2b4fe1b4b97f5c98673594 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:37.090914 18645 consensus_queue.cc:1035] T 0878ccb6135f4485b5ba35ae67a76d0c P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:56:37.091203 18068 leader_election.cc:288] T dbcdf769be2b4fe1b4b97f5c98673594 P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951), 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153)
I20240503 13:56:37.091462 15625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbcdf769be2b4fe1b4b97f5c98673594" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d" is_pre_election: true
I20240503 13:56:37.091619 15625 raft_consensus.cc:2454] T dbcdf769be2b4fe1b4b97f5c98673594 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 0.
I20240503 13:56:37.091665 15493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbcdf769be2b4fe1b4b97f5c98673594" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546" is_pre_election: true
I20240503 13:56:37.091791 15493 raft_consensus.cc:2454] T dbcdf769be2b4fe1b4b97f5c98673594 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 0.
I20240503 13:56:37.091989 15822 leader_election.cc:302] T dbcdf769be2b4fe1b4b97f5c98673594 P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:37.092144 18068 raft_consensus.cc:2781] T dbcdf769be2b4fe1b4b97f5c98673594 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:37.092202 18068 raft_consensus.cc:482] T dbcdf769be2b4fe1b4b97f5c98673594 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:37.092244 18068 raft_consensus.cc:3037] T dbcdf769be2b4fe1b4b97f5c98673594 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:37.093322 18068 raft_consensus.cc:504] T dbcdf769be2b4fe1b4b97f5c98673594 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:37.093652 18068 leader_election.cc:288] T dbcdf769be2b4fe1b4b97f5c98673594 P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 election: Requested vote from peers 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951), 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153)
I20240503 13:56:37.093923 15625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbcdf769be2b4fe1b4b97f5c98673594" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d"
I20240503 13:56:37.094019 15625 raft_consensus.cc:3037] T dbcdf769be2b4fe1b4b97f5c98673594 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:37.094108 15493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbcdf769be2b4fe1b4b97f5c98673594" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546"
I20240503 13:56:37.094033 18933 raft_consensus.cc:2932] T 0878ccb6135f4485b5ba35ae67a76d0c P 76c386bef61a4bd5a907266a5118ffe2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:37.094206 15493 raft_consensus.cc:3037] T dbcdf769be2b4fe1b4b97f5c98673594 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:37.095016 15625 raft_consensus.cc:2454] T dbcdf769be2b4fe1b4b97f5c98673594 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 1.
I20240503 13:56:37.095141 15493 raft_consensus.cc:2454] T dbcdf769be2b4fe1b4b97f5c98673594 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 1.
I20240503 13:56:37.095340 15822 leader_election.cc:302] T dbcdf769be2b4fe1b4b97f5c98673594 P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:37.095516 18068 raft_consensus.cc:2781] T dbcdf769be2b4fe1b4b97f5c98673594 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:37.095589 18068 raft_consensus.cc:686] T dbcdf769be2b4fe1b4b97f5c98673594 P e47105a0f46e4511b7112e571688a4c7 [term 1 LEADER]: Becoming Leader. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Running, Role: LEADER
I20240503 13:56:37.095703 18068 consensus_queue.cc:237] T dbcdf769be2b4fe1b4b97f5c98673594 P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:37.095949 13051 raft_consensus.cc:2932] T 0878ccb6135f4485b5ba35ae67a76d0c P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:37.096005 13447 raft_consensus.cc:2932] T 0878ccb6135f4485b5ba35ae67a76d0c P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:37.098695 12912 catalog_manager.cc:5484] T dbcdf769be2b4fe1b4b97f5c98673594 P e47105a0f46e4511b7112e571688a4c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e47105a0f46e4511b7112e571688a4c7 (127.6.98.72). New cstate: current_term: 1 leader_uuid: "e47105a0f46e4511b7112e571688a4c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:37.098692 12906 catalog_manager.cc:5484] T 0878ccb6135f4485b5ba35ae67a76d0c P 7690ee79577c45e3a2c8f8b552c0bff8 reported cstate change: config changed from index -1 to 2, VOTER ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) added. New cstate: current_term: 1 leader_uuid: "76c386bef61a4bd5a907266a5118ffe2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:37.125192 13316 consensus_queue.cc:237] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:37.126617 13051 raft_consensus.cc:1261] T d50d05d432e84d5eb0e54d1a68cdf158 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:56:37.127060 13183 raft_consensus.cc:1261] T d50d05d432e84d5eb0e54d1a68cdf158 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:56:37.127053 18914 consensus_queue.cc:1035] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:37.127394 18873 consensus_queue.cc:1035] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 13:56:37.127566 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d50d05d432e84d5eb0e54d1a68cdf158. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:37.132642 18873 raft_consensus.cc:2932] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:37.135200 13183 raft_consensus.cc:2932] T d50d05d432e84d5eb0e54d1a68cdf158 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:37.136179 12903 catalog_manager.cc:5484] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 reported cstate change: config changed from index -1 to 4, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) added. New cstate: current_term: 1 leader_uuid: "ba0194098aa84449ae17e8a0d4c43703" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:37.136919 13051 raft_consensus.cc:2932] T d50d05d432e84d5eb0e54d1a68cdf158 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:37.156724 13051 consensus_queue.cc:237] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:37.158207 15889 raft_consensus.cc:1261] T 38596c626067458495846931e0ecb7eb P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Refusing update from remote peer 60ba3ffe3b7746469834cd91c84be14e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:37.158401 15758 raft_consensus.cc:1261] T 38596c626067458495846931e0ecb7eb P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Refusing update from remote peer 60ba3ffe3b7746469834cd91c84be14e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 13:56:37.158437 12990 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153): Couldn't send request to peer 61b887f0d40d497badeca5604c1f0546. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 38596c626067458495846931e0ecb7eb. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:37.158610 18677 consensus_queue.cc:1035] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:56:37.158787 18974 consensus_queue.cc:1035] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:56:37.160738 18974 raft_consensus.cc:2932] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 61b887f0d40d497badeca5604c1f0546 (127.6.98.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:37.161391 15758 raft_consensus.cc:2932] T 38596c626067458495846931e0ecb7eb P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 61b887f0d40d497badeca5604c1f0546 (127.6.98.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:37.162817 15889 raft_consensus.cc:2932] T 38596c626067458495846931e0ecb7eb P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 61b887f0d40d497badeca5604c1f0546 (127.6.98.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:37.164168 12908 catalog_manager.cc:5484] T 38596c626067458495846931e0ecb7eb P 76c386bef61a4bd5a907266a5118ffe2 reported cstate change: config changed from index -1 to 2, VOTER 61b887f0d40d497badeca5604c1f0546 (127.6.98.69) added. New cstate: current_term: 1 leader_uuid: "60ba3ffe3b7746469834cd91c84be14e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:37.188737 15493 consensus_queue.cc:237] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:37.190202 15889 raft_consensus.cc:1261] T 27d5ced54d2e4aa9b6ea01fe8700443d P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Refusing update from remote peer 61b887f0d40d497badeca5604c1f0546: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:37.190202 13316 raft_consensus.cc:1261] T 27d5ced54d2e4aa9b6ea01fe8700443d P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Refusing update from remote peer 61b887f0d40d497badeca5604c1f0546: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 13:56:37.190409 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109): Couldn't send request to peer 7690ee79577c45e3a2c8f8b552c0bff8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 27d5ced54d2e4aa9b6ea01fe8700443d. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:37.190711 17385 consensus_queue.cc:1035] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:56:37.191850 17385 consensus_queue.cc:1035] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:56:37.193346 18771 consensus_queue.cc:1035] T d36f17b60df04c2f8965c6cdccfe1204 P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:37.193802 18787 raft_consensus.cc:2932] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } }
I20240503 13:56:37.194191 15889 raft_consensus.cc:2932] T 27d5ced54d2e4aa9b6ea01fe8700443d P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } }
I20240503 13:56:37.196681 13316 raft_consensus.cc:2932] T 27d5ced54d2e4aa9b6ea01fe8700443d P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } }
I20240503 13:56:37.197008 12903 catalog_manager.cc:5484] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 reported cstate change: config changed from index -1 to 2, VOTER 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68) added. New cstate: current_term: 1 leader_uuid: "61b887f0d40d497badeca5604c1f0546" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } }
I20240503 13:56:37.198907 18186 consensus_queue.cc:1035] T d36f17b60df04c2f8965c6cdccfe1204 P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:37.219604 13183 consensus_queue.cc:237] T 477c33f203f649019eff2b28524fad90 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:37.221249 13316 raft_consensus.cc:1261] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Refusing update from remote peer 8ccf5a4ac450414c9d3579bd5800b770: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 13:56:37.221501 13118 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 477c33f203f649019eff2b28524fad90. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:37.221541 13447 raft_consensus.cc:1261] T 477c33f203f649019eff2b28524fad90 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer 8ccf5a4ac450414c9d3579bd5800b770: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:56:37.221731 18358 consensus_queue.cc:1035] T 477c33f203f649019eff2b28524fad90 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:37.221920 19016 consensus_queue.cc:1035] T 477c33f203f649019eff2b28524fad90 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:37.225332 18358 raft_consensus.cc:2932] T 477c33f203f649019eff2b28524fad90 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:37.225786 13316 raft_consensus.cc:2932] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:37.228255 13447 raft_consensus.cc:2932] T 477c33f203f649019eff2b28524fad90 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:37.228453 12912 catalog_manager.cc:5484] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 reported cstate change: config changed from index -1 to 4, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) added. New cstate: current_term: 1 leader_uuid: "8ccf5a4ac450414c9d3579bd5800b770" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:37.233361 12912 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:56:37.237583 15493 raft_consensus.cc:1261] T dbcdf769be2b4fe1b4b97f5c98673594 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Refusing update from remote peer e47105a0f46e4511b7112e571688a4c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:37.237965 15625 raft_consensus.cc:1261] T dbcdf769be2b4fe1b4b97f5c98673594 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Refusing update from remote peer e47105a0f46e4511b7112e571688a4c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:37.238137 17953 consensus_queue.cc:1035] T dbcdf769be2b4fe1b4b97f5c98673594 P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:37.239305 17953 consensus_queue.cc:1035] T dbcdf769be2b4fe1b4b97f5c98673594 P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:37.240406 12912 catalog_manager.cc:5576] TS 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153) does not have the latest schema for tablet dbcdf769be2b4fe1b4b97f5c98673594 (table rebalancer_dynamic_table [id=2c10732c24b04c59b89993df2621459d]). Expected version 1 got 0
I20240503 13:56:37.243438 12912 catalog_manager.cc:5576] TS 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951) does not have the latest schema for tablet dbcdf769be2b4fe1b4b97f5c98673594 (table rebalancer_dynamic_table [id=2c10732c24b04c59b89993df2621459d]). Expected version 1 got 0
I20240503 13:56:37.243953 19033 tablet.cc:1722] T dbcdf769be2b4fe1b4b97f5c98673594 P e47105a0f46e4511b7112e571688a4c7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:37.244354 19033 tablet.cc:1620] T dbcdf769be2b4fe1b4b97f5c98673594 P e47105a0f46e4511b7112e571688a4c7: MemRowSet was empty: no flush needed.
I20240503 13:56:37.245466 19035 tablet.cc:1722] T dbcdf769be2b4fe1b4b97f5c98673594 P 61b887f0d40d497badeca5604c1f0546: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:37.245849 19035 tablet.cc:1620] T dbcdf769be2b4fe1b4b97f5c98673594 P 61b887f0d40d497badeca5604c1f0546: MemRowSet was empty: no flush needed.
I20240503 13:56:37.246429 19036 tablet.cc:1722] T dbcdf769be2b4fe1b4b97f5c98673594 P 009f849ee7b54c80a3f5264695047e4d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:37.246776 19036 tablet.cc:1620] T dbcdf769be2b4fe1b4b97f5c98673594 P 009f849ee7b54c80a3f5264695047e4d: MemRowSet was empty: no flush needed.
I20240503 13:56:37.246850 12908 catalog_manager.cc:5576] TS 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153) does not have the latest schema for tablet dbcdf769be2b4fe1b4b97f5c98673594 (table rebalancer_dynamic_table [id=2c10732c24b04c59b89993df2621459d]). Expected version 1 got 0
I20240503 13:56:37.247376 12906 catalog_manager.cc:5576] TS 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951) does not have the latest schema for tablet dbcdf769be2b4fe1b4b97f5c98673594 (table rebalancer_dynamic_table [id=2c10732c24b04c59b89993df2621459d]). Expected version 1 got 0
I20240503 13:56:37.254403 12906 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=2c10732c24b04c59b89993df2621459d] alter complete (version 1)
I20240503 13:56:37.260636 13183 consensus_queue.cc:237] T a49b6f596d724b35b757ee69fe15e5a5 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:37.261952 13316 raft_consensus.cc:1261] T a49b6f596d724b35b757ee69fe15e5a5 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Refusing update from remote peer 8ccf5a4ac450414c9d3579bd5800b770: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:56:37.262161 13051 raft_consensus.cc:1261] T a49b6f596d724b35b757ee69fe15e5a5 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Refusing update from remote peer 8ccf5a4ac450414c9d3579bd5800b770: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:56:37.262346 19016 consensus_queue.cc:1035] T a49b6f596d724b35b757ee69fe15e5a5 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:37.262542 19019 consensus_queue.cc:1035] T a49b6f596d724b35b757ee69fe15e5a5 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 13:56:37.262749 13121 consensus_peers.cc:485] T a49b6f596d724b35b757ee69fe15e5a5 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153): Couldn't send request to peer 61b887f0d40d497badeca5604c1f0546. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a49b6f596d724b35b757ee69fe15e5a5. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:37.264401 18857 raft_consensus.cc:2932] T a49b6f596d724b35b757ee69fe15e5a5 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 61b887f0d40d497badeca5604c1f0546 (127.6.98.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:37.264890 13051 raft_consensus.cc:2932] T a49b6f596d724b35b757ee69fe15e5a5 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 61b887f0d40d497badeca5604c1f0546 (127.6.98.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:37.266505 13316 raft_consensus.cc:2932] T a49b6f596d724b35b757ee69fe15e5a5 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 61b887f0d40d497badeca5604c1f0546 (127.6.98.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:37.267738 12903 catalog_manager.cc:5484] T a49b6f596d724b35b757ee69fe15e5a5 P 8ccf5a4ac450414c9d3579bd5800b770 reported cstate change: config changed from index -1 to 4, VOTER 61b887f0d40d497badeca5604c1f0546 (127.6.98.69) added. New cstate: current_term: 1 leader_uuid: "8ccf5a4ac450414c9d3579bd5800b770" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:37.285905 18374 consensus_queue.cc:1035] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:37.287248 13183 consensus_queue.cc:237] T b82134ddae61479eb2ced348d7558799 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:37.289418 13316 raft_consensus.cc:1261] T b82134ddae61479eb2ced348d7558799 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Refusing update from remote peer 8ccf5a4ac450414c9d3579bd5800b770: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:56:37.289814 13447 raft_consensus.cc:1261] T b82134ddae61479eb2ced348d7558799 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer 8ccf5a4ac450414c9d3579bd5800b770: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:56:37.290210 19016 consensus_queue.cc:1035] T b82134ddae61479eb2ced348d7558799 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:37.290405 18855 consensus_queue.cc:1035] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 13:56:37.290452 13121 consensus_peers.cc:485] T b82134ddae61479eb2ced348d7558799 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885): Couldn't send request to peer 76c386bef61a4bd5a907266a5118ffe2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b82134ddae61479eb2ced348d7558799. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:37.290647 19016 consensus_queue.cc:1035] T b82134ddae61479eb2ced348d7558799 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:37.293637 19019 raft_consensus.cc:2932] T b82134ddae61479eb2ced348d7558799 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:37.293977 13316 raft_consensus.cc:2932] T b82134ddae61479eb2ced348d7558799 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:37.296671 12906 catalog_manager.cc:5484] T b82134ddae61479eb2ced348d7558799 P ba0194098aa84449ae17e8a0d4c43703 reported cstate change: config changed from index -1 to 4, VOTER 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71) added. New cstate: current_term: 1 leader_uuid: "8ccf5a4ac450414c9d3579bd5800b770" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:37.296674 13445 raft_consensus.cc:2932] T b82134ddae61479eb2ced348d7558799 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:37.312644 18186 consensus_queue.cc:1035] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:37.317216 18186 consensus_queue.cc:1035] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:37.356822 12909 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:56:37.363974 19033 tablet.cc:1722] T dbcdf769be2b4fe1b4b97f5c98673594 P e47105a0f46e4511b7112e571688a4c7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:37.364333 19033 tablet.cc:1620] T dbcdf769be2b4fe1b4b97f5c98673594 P e47105a0f46e4511b7112e571688a4c7: MemRowSet was empty: no flush needed.
I20240503 13:56:37.365074 19035 tablet.cc:1722] T dbcdf769be2b4fe1b4b97f5c98673594 P 61b887f0d40d497badeca5604c1f0546: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:37.365384 19035 tablet.cc:1620] T dbcdf769be2b4fe1b4b97f5c98673594 P 61b887f0d40d497badeca5604c1f0546: MemRowSet was empty: no flush needed.
I20240503 13:56:37.366909 19036 tablet.cc:1722] T dbcdf769be2b4fe1b4b97f5c98673594 P 009f849ee7b54c80a3f5264695047e4d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:37.367216 19036 tablet.cc:1620] T dbcdf769be2b4fe1b4b97f5c98673594 P 009f849ee7b54c80a3f5264695047e4d: MemRowSet was empty: no flush needed.
I20240503 13:56:37.371976 12909 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=2c10732c24b04c59b89993df2621459d] alter complete (version 2)
I20240503 13:56:37.406561 13034 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:37.408030 12909 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_extra_table_22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:56:37.409817 15605 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:37.411881 15472 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:56:37.413223 13164 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:37.417132 15738 tablet_service.cc:1467] Processing CreateTablet for tablet 25d41dbdacc341ee93c86f7a7f056123 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=fdf22baab48647bbb6f53c78f739709c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:37.417450 15738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25d41dbdacc341ee93c86f7a7f056123. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:37.419744 19121 tablet_bootstrap.cc:492] T 25d41dbdacc341ee93c86f7a7f056123 P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:37.420953 13295 tablet_service.cc:1467] Processing CreateTablet for tablet a5a875bb0dce46f5883c302ea1729430 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=fdf22baab48647bbb6f53c78f739709c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:37.420950 15736 tablet_service.cc:1467] Processing CreateTablet for tablet e9849f54796d477db114fa7a85257fe2 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=fdf22baab48647bbb6f53c78f739709c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:37.421192 13295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5a875bb0dce46f5883c302ea1729430. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:37.421192 15736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9849f54796d477db114fa7a85257fe2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:37.423884 15605 tablet_service.cc:1467] Processing CreateTablet for tablet e9849f54796d477db114fa7a85257fe2 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=fdf22baab48647bbb6f53c78f739709c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:37.423916 15868 tablet_service.cc:1467] Processing CreateTablet for tablet a5a875bb0dce46f5883c302ea1729430 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=fdf22baab48647bbb6f53c78f739709c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:37.424181 15868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5a875bb0dce46f5883c302ea1729430. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:37.424186 15605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9849f54796d477db114fa7a85257fe2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:37.424645 15473 tablet_service.cc:1467] Processing CreateTablet for tablet e9849f54796d477db114fa7a85257fe2 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=fdf22baab48647bbb6f53c78f739709c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:37.424883 15473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9849f54796d477db114fa7a85257fe2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:37.425056 15471 tablet_service.cc:1467] Processing CreateTablet for tablet 25d41dbdacc341ee93c86f7a7f056123 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=fdf22baab48647bbb6f53c78f739709c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:37.425312 15471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25d41dbdacc341ee93c86f7a7f056123. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:37.426510 15867 tablet_service.cc:1467] Processing CreateTablet for tablet 25d41dbdacc341ee93c86f7a7f056123 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=fdf22baab48647bbb6f53c78f739709c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:37.426591 18761 tablet_bootstrap.cc:492] T e9849f54796d477db114fa7a85257fe2 P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:37.426734 15867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25d41dbdacc341ee93c86f7a7f056123. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:37.426898 18615 tablet_bootstrap.cc:492] T e9849f54796d477db114fa7a85257fe2 P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:37.427485 18761 tablet_bootstrap.cc:654] T e9849f54796d477db114fa7a85257fe2 P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:37.427696 18615 tablet_bootstrap.cc:654] T e9849f54796d477db114fa7a85257fe2 P 61b887f0d40d497badeca5604c1f0546: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:37.428416 19121 tablet_bootstrap.cc:654] T 25d41dbdacc341ee93c86f7a7f056123 P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:37.428393 15472 tablet_service.cc:1467] Processing CreateTablet for tablet a5a875bb0dce46f5883c302ea1729430 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=fdf22baab48647bbb6f53c78f739709c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:37.428560 18755 tablet_bootstrap.cc:492] T a5a875bb0dce46f5883c302ea1729430 P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:37.428630 15472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5a875bb0dce46f5883c302ea1729430. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:37.429415 18755 tablet_bootstrap.cc:654] T a5a875bb0dce46f5883c302ea1729430 P e47105a0f46e4511b7112e571688a4c7: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:37.429622 18761 tablet_bootstrap.cc:492] T e9849f54796d477db114fa7a85257fe2 P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:37.429701 18761 ts_tablet_manager.cc:1367] T e9849f54796d477db114fa7a85257fe2 P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:56:37.430130 18761 raft_consensus.cc:348] T e9849f54796d477db114fa7a85257fe2 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:37.430231 18761 raft_consensus.cc:374] T e9849f54796d477db114fa7a85257fe2 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:37.430260 18761 raft_consensus.cc:729] T e9849f54796d477db114fa7a85257fe2 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:37.430377 18761 consensus_queue.cc:260] T e9849f54796d477db114fa7a85257fe2 P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:37.430675 18761 ts_tablet_manager.cc:1398] T e9849f54796d477db114fa7a85257fe2 P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:37.432140 19121 tablet_bootstrap.cc:492] T 25d41dbdacc341ee93c86f7a7f056123 P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:37.432281 19121 ts_tablet_manager.cc:1367] T 25d41dbdacc341ee93c86f7a7f056123 P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 13:56:37.432670 13424 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 13:56:37.432704 18755 tablet_bootstrap.cc:492] T a5a875bb0dce46f5883c302ea1729430 P e47105a0f46e4511b7112e571688a4c7: No bootstrap required, opened a new log
I20240503 13:56:37.432783 18755 ts_tablet_manager.cc:1367] T a5a875bb0dce46f5883c302ea1729430 P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:56:37.432885 19121 raft_consensus.cc:348] T 25d41dbdacc341ee93c86f7a7f056123 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:37.433046 19121 raft_consensus.cc:374] T 25d41dbdacc341ee93c86f7a7f056123 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:37.433125 19121 raft_consensus.cc:729] T 25d41dbdacc341ee93c86f7a7f056123 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:37.433233 18755 raft_consensus.cc:348] T a5a875bb0dce46f5883c302ea1729430 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:37.433336 18755 raft_consensus.cc:374] T a5a875bb0dce46f5883c302ea1729430 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:37.433290 19121 consensus_queue.cc:260] T 25d41dbdacc341ee93c86f7a7f056123 P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:37.433363 18755 raft_consensus.cc:729] T a5a875bb0dce46f5883c302ea1729430 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:37.433465 18755 consensus_queue.cc:260] T a5a875bb0dce46f5883c302ea1729430 P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:37.433579 19121 ts_tablet_manager.cc:1398] T 25d41dbdacc341ee93c86f7a7f056123 P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:37.433720 18755 ts_tablet_manager.cc:1398] T a5a875bb0dce46f5883c302ea1729430 P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:37.433853 18755 tablet_bootstrap.cc:492] T 25d41dbdacc341ee93c86f7a7f056123 P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:37.434831 19121 tablet_bootstrap.cc:492] T e9849f54796d477db114fa7a85257fe2 P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:37.436039 19121 tablet_bootstrap.cc:654] T e9849f54796d477db114fa7a85257fe2 P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:37.436038 18755 tablet_bootstrap.cc:654] T 25d41dbdacc341ee93c86f7a7f056123 P e47105a0f46e4511b7112e571688a4c7: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:37.437287 19121 tablet_bootstrap.cc:492] T e9849f54796d477db114fa7a85257fe2 P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:37.437363 19121 ts_tablet_manager.cc:1367] T e9849f54796d477db114fa7a85257fe2 P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:56:37.437763 19121 raft_consensus.cc:348] T e9849f54796d477db114fa7a85257fe2 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:37.437852 19121 raft_consensus.cc:374] T e9849f54796d477db114fa7a85257fe2 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:37.437880 19121 raft_consensus.cc:729] T e9849f54796d477db114fa7a85257fe2 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:37.437992 19121 consensus_queue.cc:260] T e9849f54796d477db114fa7a85257fe2 P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:37.438246 19121 ts_tablet_manager.cc:1398] T e9849f54796d477db114fa7a85257fe2 P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:37.438706 18755 tablet_bootstrap.cc:492] T 25d41dbdacc341ee93c86f7a7f056123 P e47105a0f46e4511b7112e571688a4c7: No bootstrap required, opened a new log
I20240503 13:56:37.438792 18755 ts_tablet_manager.cc:1367] T 25d41dbdacc341ee93c86f7a7f056123 P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240503 13:56:37.439181 18755 raft_consensus.cc:348] T 25d41dbdacc341ee93c86f7a7f056123 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:37.439267 18755 raft_consensus.cc:374] T 25d41dbdacc341ee93c86f7a7f056123 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:37.439291 18755 raft_consensus.cc:729] T 25d41dbdacc341ee93c86f7a7f056123 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:37.439395 18755 consensus_queue.cc:260] T 25d41dbdacc341ee93c86f7a7f056123 P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:37.439620 18755 ts_tablet_manager.cc:1398] T 25d41dbdacc341ee93c86f7a7f056123 P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:37.443312 18615 tablet_bootstrap.cc:492] T e9849f54796d477db114fa7a85257fe2 P 61b887f0d40d497badeca5604c1f0546: No bootstrap required, opened a new log
I20240503 13:56:37.443403 18615 ts_tablet_manager.cc:1367] T e9849f54796d477db114fa7a85257fe2 P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.001s
I20240503 13:56:37.443105 19122 tablet_bootstrap.cc:492] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:37.443843 18615 raft_consensus.cc:348] T e9849f54796d477db114fa7a85257fe2 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:37.443964 18615 raft_consensus.cc:374] T e9849f54796d477db114fa7a85257fe2 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:37.443991 18615 raft_consensus.cc:729] T e9849f54796d477db114fa7a85257fe2 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:37.444103 18615 consensus_queue.cc:260] T e9849f54796d477db114fa7a85257fe2 P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:37.444336 18615 ts_tablet_manager.cc:1398] T e9849f54796d477db114fa7a85257fe2 P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:37.444473 18615 tablet_bootstrap.cc:492] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:37.445403 18615 tablet_bootstrap.cc:654] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:37.446862 13295 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:56:37.452234 19122 tablet_bootstrap.cc:654] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:37.454489 15736 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:56:37.464478 15867 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 13:56:37.478106 18615 tablet_bootstrap.cc:492] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546: No bootstrap required, opened a new log
I20240503 13:56:37.478231 18615 ts_tablet_manager.cc:1367] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.034s user 0.000s sys 0.001s
I20240503 13:56:37.479099 18615 raft_consensus.cc:348] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:37.479250 18615 raft_consensus.cc:374] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:37.479283 18615 raft_consensus.cc:729] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:37.479400 18615 consensus_queue.cc:260] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:37.479718 18615 ts_tablet_manager.cc:1398] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240503 13:56:37.479907 18615 tablet_bootstrap.cc:492] T a5a875bb0dce46f5883c302ea1729430 P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:37.481724 18615 tablet_bootstrap.cc:654] T a5a875bb0dce46f5883c302ea1729430 P 61b887f0d40d497badeca5604c1f0546: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:37.483166 19032 raft_consensus.cc:482] T a5a875bb0dce46f5883c302ea1729430 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:37.483269 19032 raft_consensus.cc:504] T a5a875bb0dce46f5883c302ea1729430 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:37.483675 19032 leader_election.cc:288] T a5a875bb0dce46f5883c302ea1729430 P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153), ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:37.484620 13314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5a875bb0dce46f5883c302ea1729430" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703" is_pre_election: true
I20240503 13:56:37.485036 19122 tablet_bootstrap.cc:492] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:37.485033 15493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5a875bb0dce46f5883c302ea1729430" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546" is_pre_election: true
I20240503 13:56:37.485136 19122 ts_tablet_manager.cc:1367] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.042s user 0.002s sys 0.000s
I20240503 13:56:37.485587 19122 raft_consensus.cc:348] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:37.485688 19122 raft_consensus.cc:374] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:37.485720 19122 raft_consensus.cc:729] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:37.485827 19122 consensus_queue.cc:260] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
W20240503 13:56:37.485927 15825 leader_election.cc:341] T a5a875bb0dce46f5883c302ea1729430 P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279): Illegal state: must be running to vote when last-logged opid is not known
W20240503 13:56:37.486042 15825 leader_election.cc:341] T a5a875bb0dce46f5883c302ea1729430 P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153): Illegal state: must be running to vote when last-logged opid is not known
I20240503 13:56:37.486073 19122 ts_tablet_manager.cc:1398] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:37.486078 15825 leader_election.cc:302] T a5a875bb0dce46f5883c302ea1729430 P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e47105a0f46e4511b7112e571688a4c7; no voters: 61b887f0d40d497badeca5604c1f0546, ba0194098aa84449ae17e8a0d4c43703
I20240503 13:56:37.486289 12911 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:56:37.487049 19032 raft_consensus.cc:2726] T a5a875bb0dce46f5883c302ea1729430 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 13:56:37.488461 18615 tablet_bootstrap.cc:492] T a5a875bb0dce46f5883c302ea1729430 P 61b887f0d40d497badeca5604c1f0546: No bootstrap required, opened a new log
I20240503 13:56:37.488581 18615 ts_tablet_manager.cc:1367] T a5a875bb0dce46f5883c302ea1729430 P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.002s
I20240503 13:56:37.489014 18615 raft_consensus.cc:348] T a5a875bb0dce46f5883c302ea1729430 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:37.489117 18615 raft_consensus.cc:374] T a5a875bb0dce46f5883c302ea1729430 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:37.489148 18615 raft_consensus.cc:729] T a5a875bb0dce46f5883c302ea1729430 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:37.489261 18615 consensus_queue.cc:260] T a5a875bb0dce46f5883c302ea1729430 P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:37.489537 18615 ts_tablet_manager.cc:1398] T a5a875bb0dce46f5883c302ea1729430 P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:37.492460 19033 tablet.cc:1722] T dbcdf769be2b4fe1b4b97f5c98673594 P e47105a0f46e4511b7112e571688a4c7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:37.493631 19036 tablet.cc:1722] T dbcdf769be2b4fe1b4b97f5c98673594 P 009f849ee7b54c80a3f5264695047e4d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:37.498025 19035 tablet.cc:1722] T dbcdf769be2b4fe1b4b97f5c98673594 P 61b887f0d40d497badeca5604c1f0546: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:37.498353 12909 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table_new_name [id=2c10732c24b04c59b89993df2621459d] alter complete (version 3)
I20240503 13:56:37.509261 19135 ts_tablet_manager.cc:902] T d50d05d432e84d5eb0e54d1a68cdf158 P e47105a0f46e4511b7112e571688a4c7: Initiating tablet copy from peer ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:37.509797 19135 tablet_copy_client.cc:308] T d50d05d432e84d5eb0e54d1a68cdf158 P e47105a0f46e4511b7112e571688a4c7: tablet copy: Beginning tablet copy session from remote peer at address 127.6.98.67:42279
I20240503 13:56:37.510238 13336 tablet_copy_service.cc:140] P ba0194098aa84449ae17e8a0d4c43703: Received BeginTabletCopySession request for tablet d50d05d432e84d5eb0e54d1a68cdf158 from peer e47105a0f46e4511b7112e571688a4c7 ({username='slave'} at 127.6.98.72:58879)
I20240503 13:56:37.510347 13336 tablet_copy_service.cc:161] P ba0194098aa84449ae17e8a0d4c43703: Beginning new tablet copy session on tablet d50d05d432e84d5eb0e54d1a68cdf158 from peer e47105a0f46e4511b7112e571688a4c7 at {username='slave'} at 127.6.98.72:58879: session id = e47105a0f46e4511b7112e571688a4c7-d50d05d432e84d5eb0e54d1a68cdf158
I20240503 13:56:37.511162 13336 tablet_copy_source_session.cc:205] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:56:37.511974 19135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d50d05d432e84d5eb0e54d1a68cdf158. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:37.517163 19135 tablet_copy_client.cc:783] T d50d05d432e84d5eb0e54d1a68cdf158 P e47105a0f46e4511b7112e571688a4c7: tablet copy: Starting download of 0 data blocks...
I20240503 13:56:37.517340 19135 tablet_copy_client.cc:647] T d50d05d432e84d5eb0e54d1a68cdf158 P e47105a0f46e4511b7112e571688a4c7: tablet copy: Starting download of 1 WAL segments...
I20240503 13:56:37.518573 19135 tablet_copy_client.cc:523] T d50d05d432e84d5eb0e54d1a68cdf158 P e47105a0f46e4511b7112e571688a4c7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:56:37.519738 19135 tablet_bootstrap.cc:492] T d50d05d432e84d5eb0e54d1a68cdf158 P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:37.540243 19135 tablet_bootstrap.cc:492] T d50d05d432e84d5eb0e54d1a68cdf158 P e47105a0f46e4511b7112e571688a4c7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:56:37.540591 19135 tablet_bootstrap.cc:492] T d50d05d432e84d5eb0e54d1a68cdf158 P e47105a0f46e4511b7112e571688a4c7: Bootstrap complete.
I20240503 13:56:37.540779 19135 ts_tablet_manager.cc:1367] T d50d05d432e84d5eb0e54d1a68cdf158 P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.021s user 0.021s sys 0.001s
I20240503 13:56:37.541193 19135 raft_consensus.cc:348] T d50d05d432e84d5eb0e54d1a68cdf158 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:37.541321 19135 raft_consensus.cc:729] T d50d05d432e84d5eb0e54d1a68cdf158 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:37.541461 19135 consensus_queue.cc:260] T d50d05d432e84d5eb0e54d1a68cdf158 P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:37.541663 18752 raft_consensus.cc:482] T e9849f54796d477db114fa7a85257fe2 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:37.541765 18752 raft_consensus.cc:504] T e9849f54796d477db114fa7a85257fe2 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:37.541849 19135 ts_tablet_manager.cc:1398] T d50d05d432e84d5eb0e54d1a68cdf158 P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:37.542210 18752 leader_election.cc:288] T e9849f54796d477db114fa7a85257fe2 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951), 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153)
I20240503 13:56:37.542770 13336 tablet_copy_service.cc:342] P ba0194098aa84449ae17e8a0d4c43703: Request end of tablet copy session e47105a0f46e4511b7112e571688a4c7-d50d05d432e84d5eb0e54d1a68cdf158 received from {username='slave'} at 127.6.98.72:58879
I20240503 13:56:37.542860 13336 tablet_copy_service.cc:433] P ba0194098aa84449ae17e8a0d4c43703: ending tablet copy session e47105a0f46e4511b7112e571688a4c7-d50d05d432e84d5eb0e54d1a68cdf158 on tablet d50d05d432e84d5eb0e54d1a68cdf158 with peer e47105a0f46e4511b7112e571688a4c7
I20240503 13:56:37.542809 15625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9849f54796d477db114fa7a85257fe2" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d" is_pre_election: true
I20240503 13:56:37.542812 15493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9849f54796d477db114fa7a85257fe2" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546" is_pre_election: true
I20240503 13:56:37.542941 15493 raft_consensus.cc:2454] T e9849f54796d477db114fa7a85257fe2 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 0.
I20240503 13:56:37.542941 15625 raft_consensus.cc:2454] T e9849f54796d477db114fa7a85257fe2 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 0.
I20240503 13:56:37.543215 15694 leader_election.cc:302] T e9849f54796d477db114fa7a85257fe2 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61b887f0d40d497badeca5604c1f0546, 76c386bef61a4bd5a907266a5118ffe2; no voters:
I20240503 13:56:37.543498 18752 raft_consensus.cc:2781] T e9849f54796d477db114fa7a85257fe2 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:37.543546 18752 raft_consensus.cc:482] T e9849f54796d477db114fa7a85257fe2 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:37.543582 18752 raft_consensus.cc:3037] T e9849f54796d477db114fa7a85257fe2 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:37.544821 18752 raft_consensus.cc:504] T e9849f54796d477db114fa7a85257fe2 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:37.545186 18752 leader_election.cc:288] T e9849f54796d477db114fa7a85257fe2 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 election: Requested vote from peers 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951), 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153)
I20240503 13:56:37.545504 15625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9849f54796d477db114fa7a85257fe2" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d"
I20240503 13:56:37.545622 15625 raft_consensus.cc:3037] T e9849f54796d477db114fa7a85257fe2 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:37.546546 15625 raft_consensus.cc:2454] T e9849f54796d477db114fa7a85257fe2 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 1.
I20240503 13:56:37.546953 15493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9849f54796d477db114fa7a85257fe2" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546"
I20240503 13:56:37.547077 15493 raft_consensus.cc:3037] T e9849f54796d477db114fa7a85257fe2 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:37.547919 15493 raft_consensus.cc:2454] T e9849f54796d477db114fa7a85257fe2 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 1.
I20240503 13:56:37.547969 15691 leader_election.cc:302] T e9849f54796d477db114fa7a85257fe2 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 76c386bef61a4bd5a907266a5118ffe2; no voters:
I20240503 13:56:37.548123 18752 raft_consensus.cc:2781] T e9849f54796d477db114fa7a85257fe2 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:37.548197 18752 raft_consensus.cc:686] T e9849f54796d477db114fa7a85257fe2 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 LEADER]: Becoming Leader. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Running, Role: LEADER
I20240503 13:56:37.548328 18752 consensus_queue.cc:237] T e9849f54796d477db114fa7a85257fe2 P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:37.550164 12909 catalog_manager.cc:5484] T e9849f54796d477db114fa7a85257fe2 P 76c386bef61a4bd5a907266a5118ffe2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71). New cstate: current_term: 1 leader_uuid: "76c386bef61a4bd5a907266a5118ffe2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:37.593704 19180 ts_tablet_manager.cc:902] T 38596c626067458495846931e0ecb7eb P 61b887f0d40d497badeca5604c1f0546: Initiating tablet copy from peer 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:37.594409 19180 tablet_copy_client.cc:308] T 38596c626067458495846931e0ecb7eb P 61b887f0d40d497badeca5604c1f0546: tablet copy: Beginning tablet copy session from remote peer at address 127.6.98.65:34695
I20240503 13:56:37.595239 13074 tablet_copy_service.cc:140] P 60ba3ffe3b7746469834cd91c84be14e: Received BeginTabletCopySession request for tablet 38596c626067458495846931e0ecb7eb from peer 61b887f0d40d497badeca5604c1f0546 ({username='slave'} at 127.6.98.69:58129)
I20240503 13:56:37.595355 13074 tablet_copy_service.cc:161] P 60ba3ffe3b7746469834cd91c84be14e: Beginning new tablet copy session on tablet 38596c626067458495846931e0ecb7eb from peer 61b887f0d40d497badeca5604c1f0546 at {username='slave'} at 127.6.98.69:58129: session id = 61b887f0d40d497badeca5604c1f0546-38596c626067458495846931e0ecb7eb
I20240503 13:56:37.596360 13074 tablet_copy_source_session.cc:205] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:56:37.597352 19180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38596c626067458495846931e0ecb7eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:37.599550 19180 tablet_copy_client.cc:783] T 38596c626067458495846931e0ecb7eb P 61b887f0d40d497badeca5604c1f0546: tablet copy: Starting download of 0 data blocks...
I20240503 13:56:37.599681 19180 tablet_copy_client.cc:647] T 38596c626067458495846931e0ecb7eb P 61b887f0d40d497badeca5604c1f0546: tablet copy: Starting download of 1 WAL segments...
I20240503 13:56:37.600814 19180 tablet_copy_client.cc:523] T 38596c626067458495846931e0ecb7eb P 61b887f0d40d497badeca5604c1f0546: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:56:37.601886 19180 tablet_bootstrap.cc:492] T 38596c626067458495846931e0ecb7eb P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:37.603905 12903 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:37.604079 12903 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:37.605202 19180 tablet_bootstrap.cc:492] T 38596c626067458495846931e0ecb7eb P 61b887f0d40d497badeca5604c1f0546: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:56:37.605429 19180 tablet_bootstrap.cc:492] T 38596c626067458495846931e0ecb7eb P 61b887f0d40d497badeca5604c1f0546: Bootstrap complete.
I20240503 13:56:37.605607 19180 ts_tablet_manager.cc:1367] T 38596c626067458495846931e0ecb7eb P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20240503 13:56:37.606093 19180 raft_consensus.cc:348] T 38596c626067458495846931e0ecb7eb P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:37.606212 19180 raft_consensus.cc:729] T 38596c626067458495846931e0ecb7eb P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:37.606490 12903 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Sending DeleteTablet for 3 replicas of tablet dbcdf769be2b4fe1b4b97f5c98673594
I20240503 13:56:37.606526 19180 consensus_queue.cc:260] T 38596c626067458495846931e0ecb7eb P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:37.606823 19180 ts_tablet_manager.cc:1398] T 38596c626067458495846931e0ecb7eb P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240503 13:56:37.607070 15472 tablet_service.cc:1514] Processing DeleteTablet for tablet dbcdf769be2b4fe1b4b97f5c98673594 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:37 UTC) from {username='slave'} at 127.0.0.1:58758
I20240503 13:56:37.607380 19186 tablet_replica.cc:330] T dbcdf769be2b4fe1b4b97f5c98673594 P 61b887f0d40d497badeca5604c1f0546: stopping tablet replica
I20240503 13:56:37.607674 15605 tablet_service.cc:1514] Processing DeleteTablet for tablet dbcdf769be2b4fe1b4b97f5c98673594 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:37 UTC) from {username='slave'} at 127.0.0.1:60718
I20240503 13:56:37.607899 13074 tablet_copy_service.cc:342] P 60ba3ffe3b7746469834cd91c84be14e: Request end of tablet copy session 61b887f0d40d497badeca5604c1f0546-38596c626067458495846931e0ecb7eb received from {username='slave'} at 127.6.98.69:58129
I20240503 13:56:37.607905 15867 tablet_service.cc:1514] Processing DeleteTablet for tablet dbcdf769be2b4fe1b4b97f5c98673594 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:37 UTC) from {username='slave'} at 127.0.0.1:50498
I20240503 13:56:37.607983 13074 tablet_copy_service.cc:433] P 60ba3ffe3b7746469834cd91c84be14e: ending tablet copy session 61b887f0d40d497badeca5604c1f0546-38596c626067458495846931e0ecb7eb on tablet 38596c626067458495846931e0ecb7eb with peer 61b887f0d40d497badeca5604c1f0546
I20240503 13:56:37.607970 19186 raft_consensus.cc:2229] T dbcdf769be2b4fe1b4b97f5c98673594 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:37.608211 19186 raft_consensus.cc:2258] T dbcdf769be2b4fe1b4b97f5c98673594 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:37.608242 19188 tablet_replica.cc:330] T dbcdf769be2b4fe1b4b97f5c98673594 P e47105a0f46e4511b7112e571688a4c7: stopping tablet replica
I20240503 13:56:37.608378 19188 raft_consensus.cc:2229] T dbcdf769be2b4fe1b4b97f5c98673594 P e47105a0f46e4511b7112e571688a4c7 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:56:37.608705 19188 raft_consensus.cc:2258] T dbcdf769be2b4fe1b4b97f5c98673594 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:37.608862 19186 ts_tablet_manager.cc:1875] T dbcdf769be2b4fe1b4b97f5c98673594 P 61b887f0d40d497badeca5604c1f0546: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:37.609210 19188 ts_tablet_manager.cc:1875] T dbcdf769be2b4fe1b4b97f5c98673594 P e47105a0f46e4511b7112e571688a4c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:37.611035 19187 tablet_replica.cc:330] T dbcdf769be2b4fe1b4b97f5c98673594 P 009f849ee7b54c80a3f5264695047e4d: stopping tablet replica
I20240503 13:56:37.611183 19187 raft_consensus.cc:2229] T dbcdf769be2b4fe1b4b97f5c98673594 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:37.611411 19186 ts_tablet_manager.cc:1888] T dbcdf769be2b4fe1b4b97f5c98673594 P 61b887f0d40d497badeca5604c1f0546: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:37.611533 19186 log.cc:1196] T dbcdf769be2b4fe1b4b97f5c98673594 P 61b887f0d40d497badeca5604c1f0546: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/wal/wals/dbcdf769be2b4fe1b4b97f5c98673594
I20240503 13:56:37.611891 19186 ts_tablet_manager.cc:1909] T dbcdf769be2b4fe1b4b97f5c98673594 P 61b887f0d40d497badeca5604c1f0546: Deleting consensus metadata
I20240503 13:56:37.611923 19188 ts_tablet_manager.cc:1888] T dbcdf769be2b4fe1b4b97f5c98673594 P e47105a0f46e4511b7112e571688a4c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:37.612046 19188 log.cc:1196] T dbcdf769be2b4fe1b4b97f5c98673594 P e47105a0f46e4511b7112e571688a4c7: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-7/wal/wals/dbcdf769be2b4fe1b4b97f5c98673594
I20240503 13:56:37.612357 19188 ts_tablet_manager.cc:1909] T dbcdf769be2b4fe1b4b97f5c98673594 P e47105a0f46e4511b7112e571688a4c7: Deleting consensus metadata
I20240503 13:56:37.612653 19187 raft_consensus.cc:2258] T dbcdf769be2b4fe1b4b97f5c98673594 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:37.612682 12899 catalog_manager.cc:4830] TS 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153): tablet dbcdf769be2b4fe1b4b97f5c98673594 (table rebalancer_dynamic_table_new_name [id=2c10732c24b04c59b89993df2621459d]) successfully deleted
I20240503 13:56:37.613209 12896 catalog_manager.cc:4830] TS e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): tablet dbcdf769be2b4fe1b4b97f5c98673594 (table rebalancer_dynamic_table_new_name [id=2c10732c24b04c59b89993df2621459d]) successfully deleted
I20240503 13:56:37.613464 19189 ts_tablet_manager.cc:902] T 27d5ced54d2e4aa9b6ea01fe8700443d P 7690ee79577c45e3a2c8f8b552c0bff8: Initiating tablet copy from peer 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153)
I20240503 13:56:37.613476 19187 ts_tablet_manager.cc:1875] T dbcdf769be2b4fe1b4b97f5c98673594 P 009f849ee7b54c80a3f5264695047e4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:37.614004 19189 tablet_copy_client.cc:308] T 27d5ced54d2e4aa9b6ea01fe8700443d P 7690ee79577c45e3a2c8f8b552c0bff8: tablet copy: Beginning tablet copy session from remote peer at address 127.6.98.69:39153
I20240503 13:56:37.614468 15513 tablet_copy_service.cc:140] P 61b887f0d40d497badeca5604c1f0546: Received BeginTabletCopySession request for tablet 27d5ced54d2e4aa9b6ea01fe8700443d from peer 7690ee79577c45e3a2c8f8b552c0bff8 ({username='slave'} at 127.6.98.68:42017)
I20240503 13:56:37.614589 15513 tablet_copy_service.cc:161] P 61b887f0d40d497badeca5604c1f0546: Beginning new tablet copy session on tablet 27d5ced54d2e4aa9b6ea01fe8700443d from peer 7690ee79577c45e3a2c8f8b552c0bff8 at {username='slave'} at 127.6.98.68:42017: session id = 7690ee79577c45e3a2c8f8b552c0bff8-27d5ced54d2e4aa9b6ea01fe8700443d
I20240503 13:56:37.615983 15513 tablet_copy_source_session.cc:205] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:56:37.616078 19187 ts_tablet_manager.cc:1888] T dbcdf769be2b4fe1b4b97f5c98673594 P 009f849ee7b54c80a3f5264695047e4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:37.617082 19189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27d5ced54d2e4aa9b6ea01fe8700443d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:37.619853 19189 tablet_copy_client.cc:783] T 27d5ced54d2e4aa9b6ea01fe8700443d P 7690ee79577c45e3a2c8f8b552c0bff8: tablet copy: Starting download of 0 data blocks...
I20240503 13:56:37.619978 19187 log.cc:1196] T dbcdf769be2b4fe1b4b97f5c98673594 P 009f849ee7b54c80a3f5264695047e4d: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/wal/wals/dbcdf769be2b4fe1b4b97f5c98673594
I20240503 13:56:37.620280 19187 ts_tablet_manager.cc:1909] T dbcdf769be2b4fe1b4b97f5c98673594 P 009f849ee7b54c80a3f5264695047e4d: Deleting consensus metadata
I20240503 13:56:37.621099 12896 catalog_manager.cc:4830] TS 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951): tablet dbcdf769be2b4fe1b4b97f5c98673594 (table rebalancer_dynamic_table_new_name [id=2c10732c24b04c59b89993df2621459d]) successfully deleted
I20240503 13:56:37.622269 19189 tablet_copy_client.cc:647] T 27d5ced54d2e4aa9b6ea01fe8700443d P 7690ee79577c45e3a2c8f8b552c0bff8: tablet copy: Starting download of 1 WAL segments...
I20240503 13:56:37.622749 19135 ts_tablet_manager.cc:902] T 477c33f203f649019eff2b28524fad90 P e47105a0f46e4511b7112e571688a4c7: Initiating tablet copy from peer 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:37.623293 19135 tablet_copy_client.cc:308] T 477c33f203f649019eff2b28524fad90 P e47105a0f46e4511b7112e571688a4c7: tablet copy: Beginning tablet copy session from remote peer at address 127.6.98.66:36387
I20240503 13:56:37.623736 13205 tablet_copy_service.cc:140] P 8ccf5a4ac450414c9d3579bd5800b770: Received BeginTabletCopySession request for tablet 477c33f203f649019eff2b28524fad90 from peer e47105a0f46e4511b7112e571688a4c7 ({username='slave'} at 127.6.98.72:45181)
I20240503 13:56:37.623870 13205 tablet_copy_service.cc:161] P 8ccf5a4ac450414c9d3579bd5800b770: Beginning new tablet copy session on tablet 477c33f203f649019eff2b28524fad90 from peer e47105a0f46e4511b7112e571688a4c7 at {username='slave'} at 127.6.98.72:45181: session id = e47105a0f46e4511b7112e571688a4c7-477c33f203f649019eff2b28524fad90
I20240503 13:56:37.624918 13205 tablet_copy_source_session.cc:205] T 477c33f203f649019eff2b28524fad90 P 8ccf5a4ac450414c9d3579bd5800b770: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:56:37.625449 19189 tablet_copy_client.cc:523] T 27d5ced54d2e4aa9b6ea01fe8700443d P 7690ee79577c45e3a2c8f8b552c0bff8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:56:37.625761 19135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 477c33f203f649019eff2b28524fad90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:37.626703 19189 tablet_bootstrap.cc:492] T 27d5ced54d2e4aa9b6ea01fe8700443d P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:37.627769 19135 tablet_copy_client.cc:783] T 477c33f203f649019eff2b28524fad90 P e47105a0f46e4511b7112e571688a4c7: tablet copy: Starting download of 0 data blocks...
I20240503 13:56:37.627903 19135 tablet_copy_client.cc:647] T 477c33f203f649019eff2b28524fad90 P e47105a0f46e4511b7112e571688a4c7: tablet copy: Starting download of 1 WAL segments...
I20240503 13:56:37.629032 19135 tablet_copy_client.cc:523] T 477c33f203f649019eff2b28524fad90 P e47105a0f46e4511b7112e571688a4c7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:56:37.629993 19189 tablet_bootstrap.cc:492] T 27d5ced54d2e4aa9b6ea01fe8700443d P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:56:37.630056 19135 tablet_bootstrap.cc:492] T 477c33f203f649019eff2b28524fad90 P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:37.630214 19189 tablet_bootstrap.cc:492] T 27d5ced54d2e4aa9b6ea01fe8700443d P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap complete.
I20240503 13:56:37.630359 19189 ts_tablet_manager.cc:1367] T 27d5ced54d2e4aa9b6ea01fe8700443d P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 13:56:37.630846 19189 raft_consensus.cc:348] T 27d5ced54d2e4aa9b6ea01fe8700443d P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:37.630980 19189 raft_consensus.cc:729] T 27d5ced54d2e4aa9b6ea01fe8700443d P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:37.631139 19189 consensus_queue.cc:260] T 27d5ced54d2e4aa9b6ea01fe8700443d P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:37.631464 19189 ts_tablet_manager.cc:1398] T 27d5ced54d2e4aa9b6ea01fe8700443d P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 13:56:37.632089 15513 tablet_copy_service.cc:342] P 61b887f0d40d497badeca5604c1f0546: Request end of tablet copy session 7690ee79577c45e3a2c8f8b552c0bff8-27d5ced54d2e4aa9b6ea01fe8700443d received from {username='slave'} at 127.6.98.68:42017
I20240503 13:56:37.632164 15513 tablet_copy_service.cc:433] P 61b887f0d40d497badeca5604c1f0546: ending tablet copy session 7690ee79577c45e3a2c8f8b552c0bff8-27d5ced54d2e4aa9b6ea01fe8700443d on tablet 27d5ced54d2e4aa9b6ea01fe8700443d with peer 7690ee79577c45e3a2c8f8b552c0bff8
I20240503 13:56:37.648960 19135 tablet_bootstrap.cc:492] T 477c33f203f649019eff2b28524fad90 P e47105a0f46e4511b7112e571688a4c7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:56:37.649240 19135 tablet_bootstrap.cc:492] T 477c33f203f649019eff2b28524fad90 P e47105a0f46e4511b7112e571688a4c7: Bootstrap complete.
I20240503 13:56:37.649410 19135 ts_tablet_manager.cc:1367] T 477c33f203f649019eff2b28524fad90 P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.019s user 0.019s sys 0.000s
I20240503 13:56:37.649840 19135 raft_consensus.cc:348] T 477c33f203f649019eff2b28524fad90 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:37.649947 19135 raft_consensus.cc:729] T 477c33f203f649019eff2b28524fad90 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:37.650075 19135 consensus_queue.cc:260] T 477c33f203f649019eff2b28524fad90 P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:37.650337 19135 ts_tablet_manager.cc:1398] T 477c33f203f649019eff2b28524fad90 P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:37.650950 13205 tablet_copy_service.cc:342] P 8ccf5a4ac450414c9d3579bd5800b770: Request end of tablet copy session e47105a0f46e4511b7112e571688a4c7-477c33f203f649019eff2b28524fad90 received from {username='slave'} at 127.6.98.72:45181
I20240503 13:56:37.651031 13205 tablet_copy_service.cc:433] P 8ccf5a4ac450414c9d3579bd5800b770: ending tablet copy session e47105a0f46e4511b7112e571688a4c7-477c33f203f649019eff2b28524fad90 on tablet 477c33f203f649019eff2b28524fad90 with peer e47105a0f46e4511b7112e571688a4c7
I20240503 13:56:37.668947 19200 ts_tablet_manager.cc:902] T dc544319be0b484f8e9e5a3fbc40f530 P 76c386bef61a4bd5a907266a5118ffe2: Initiating tablet copy from peer ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:37.669482 19200 tablet_copy_client.cc:308] T dc544319be0b484f8e9e5a3fbc40f530 P 76c386bef61a4bd5a907266a5118ffe2: tablet copy: Beginning tablet copy session from remote peer at address 127.6.98.67:42279
I20240503 13:56:37.669941 13336 tablet_copy_service.cc:140] P ba0194098aa84449ae17e8a0d4c43703: Received BeginTabletCopySession request for tablet dc544319be0b484f8e9e5a3fbc40f530 from peer 76c386bef61a4bd5a907266a5118ffe2 ({username='slave'} at 127.6.98.71:53369)
I20240503 13:56:37.670053 13336 tablet_copy_service.cc:161] P ba0194098aa84449ae17e8a0d4c43703: Beginning new tablet copy session on tablet dc544319be0b484f8e9e5a3fbc40f530 from peer 76c386bef61a4bd5a907266a5118ffe2 at {username='slave'} at 127.6.98.71:53369: session id = 76c386bef61a4bd5a907266a5118ffe2-dc544319be0b484f8e9e5a3fbc40f530
I20240503 13:56:37.670787 13336 tablet_copy_source_session.cc:205] T dc544319be0b484f8e9e5a3fbc40f530 P ba0194098aa84449ae17e8a0d4c43703: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:56:37.671737 19200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc544319be0b484f8e9e5a3fbc40f530. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:37.674003 19200 tablet_copy_client.cc:783] T dc544319be0b484f8e9e5a3fbc40f530 P 76c386bef61a4bd5a907266a5118ffe2: tablet copy: Starting download of 0 data blocks...
I20240503 13:56:37.674165 19200 tablet_copy_client.cc:647] T dc544319be0b484f8e9e5a3fbc40f530 P 76c386bef61a4bd5a907266a5118ffe2: tablet copy: Starting download of 1 WAL segments...
I20240503 13:56:37.675580 19200 tablet_copy_client.cc:523] T dc544319be0b484f8e9e5a3fbc40f530 P 76c386bef61a4bd5a907266a5118ffe2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:56:37.676726 19200 tablet_bootstrap.cc:492] T dc544319be0b484f8e9e5a3fbc40f530 P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:37.692751 19206 ts_tablet_manager.cc:902] T 0878ccb6135f4485b5ba35ae67a76d0c P ba0194098aa84449ae17e8a0d4c43703: Initiating tablet copy from peer 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885)
I20240503 13:56:37.693624 19206 tablet_copy_client.cc:308] T 0878ccb6135f4485b5ba35ae67a76d0c P ba0194098aa84449ae17e8a0d4c43703: tablet copy: Beginning tablet copy session from remote peer at address 127.6.98.71:42885
I20240503 13:56:37.694270 15778 tablet_copy_service.cc:140] P 76c386bef61a4bd5a907266a5118ffe2: Received BeginTabletCopySession request for tablet 0878ccb6135f4485b5ba35ae67a76d0c from peer ba0194098aa84449ae17e8a0d4c43703 ({username='slave'} at 127.6.98.67:56005)
I20240503 13:56:37.694411 15778 tablet_copy_service.cc:161] P 76c386bef61a4bd5a907266a5118ffe2: Beginning new tablet copy session on tablet 0878ccb6135f4485b5ba35ae67a76d0c from peer ba0194098aa84449ae17e8a0d4c43703 at {username='slave'} at 127.6.98.67:56005: session id = ba0194098aa84449ae17e8a0d4c43703-0878ccb6135f4485b5ba35ae67a76d0c
I20240503 13:56:37.695566 15778 tablet_copy_source_session.cc:205] T 0878ccb6135f4485b5ba35ae67a76d0c P 76c386bef61a4bd5a907266a5118ffe2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:56:37.697309 19200 tablet_bootstrap.cc:492] T dc544319be0b484f8e9e5a3fbc40f530 P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:56:37.697366 19206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0878ccb6135f4485b5ba35ae67a76d0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:37.697551 19200 tablet_bootstrap.cc:492] T dc544319be0b484f8e9e5a3fbc40f530 P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap complete.
I20240503 13:56:37.697708 19200 ts_tablet_manager.cc:1367] T dc544319be0b484f8e9e5a3fbc40f530 P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.021s user 0.016s sys 0.003s
I20240503 13:56:37.698096 19200 raft_consensus.cc:348] T dc544319be0b484f8e9e5a3fbc40f530 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:37.698195 19200 raft_consensus.cc:729] T dc544319be0b484f8e9e5a3fbc40f530 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:37.698304 19200 consensus_queue.cc:260] T dc544319be0b484f8e9e5a3fbc40f530 P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:37.698571 19200 ts_tablet_manager.cc:1398] T dc544319be0b484f8e9e5a3fbc40f530 P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:37.699162 13336 tablet_copy_service.cc:342] P ba0194098aa84449ae17e8a0d4c43703: Request end of tablet copy session 76c386bef61a4bd5a907266a5118ffe2-dc544319be0b484f8e9e5a3fbc40f530 received from {username='slave'} at 127.6.98.71:53369
I20240503 13:56:37.699239 13336 tablet_copy_service.cc:433] P ba0194098aa84449ae17e8a0d4c43703: ending tablet copy session 76c386bef61a4bd5a907266a5118ffe2-dc544319be0b484f8e9e5a3fbc40f530 on tablet dc544319be0b484f8e9e5a3fbc40f530 with peer 76c386bef61a4bd5a907266a5118ffe2
I20240503 13:56:37.699532 19206 tablet_copy_client.cc:783] T 0878ccb6135f4485b5ba35ae67a76d0c P ba0194098aa84449ae17e8a0d4c43703: tablet copy: Starting download of 0 data blocks...
I20240503 13:56:37.699746 19206 tablet_copy_client.cc:647] T 0878ccb6135f4485b5ba35ae67a76d0c P ba0194098aa84449ae17e8a0d4c43703: tablet copy: Starting download of 1 WAL segments...
I20240503 13:56:37.701277 19206 tablet_copy_client.cc:523] T 0878ccb6135f4485b5ba35ae67a76d0c P ba0194098aa84449ae17e8a0d4c43703: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:56:37.702900 19206 tablet_bootstrap.cc:492] T 0878ccb6135f4485b5ba35ae67a76d0c P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:37.706420 19206 tablet_bootstrap.cc:492] T 0878ccb6135f4485b5ba35ae67a76d0c P ba0194098aa84449ae17e8a0d4c43703: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:56:37.706642 19206 tablet_bootstrap.cc:492] T 0878ccb6135f4485b5ba35ae67a76d0c P ba0194098aa84449ae17e8a0d4c43703: Bootstrap complete.
I20240503 13:56:37.706781 19206 ts_tablet_manager.cc:1367] T 0878ccb6135f4485b5ba35ae67a76d0c P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240503 13:56:37.707165 19206 raft_consensus.cc:348] T 0878ccb6135f4485b5ba35ae67a76d0c P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:37.707268 19206 raft_consensus.cc:729] T 0878ccb6135f4485b5ba35ae67a76d0c P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:37.707391 19206 consensus_queue.cc:260] T 0878ccb6135f4485b5ba35ae67a76d0c P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:37.707676 19206 ts_tablet_manager.cc:1398] T 0878ccb6135f4485b5ba35ae67a76d0c P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 13:56:37.708775 15778 tablet_copy_service.cc:342] P 76c386bef61a4bd5a907266a5118ffe2: Request end of tablet copy session ba0194098aa84449ae17e8a0d4c43703-0878ccb6135f4485b5ba35ae67a76d0c received from {username='slave'} at 127.6.98.67:56005
I20240503 13:56:37.708864 15778 tablet_copy_service.cc:433] P 76c386bef61a4bd5a907266a5118ffe2: ending tablet copy session ba0194098aa84449ae17e8a0d4c43703-0878ccb6135f4485b5ba35ae67a76d0c on tablet 0878ccb6135f4485b5ba35ae67a76d0c with peer ba0194098aa84449ae17e8a0d4c43703
I20240503 13:56:37.709051 12903 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:56:37.711266 19032 raft_consensus.cc:482] T 25d41dbdacc341ee93c86f7a7f056123 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:37.711371 19032 raft_consensus.cc:504] T 25d41dbdacc341ee93c86f7a7f056123 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:37.711813 19032 leader_election.cc:288] T 25d41dbdacc341ee93c86f7a7f056123 P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153), 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885)
I20240503 13:56:37.712247 15758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25d41dbdacc341ee93c86f7a7f056123" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2" is_pre_election: true
I20240503 13:56:37.712247 15493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25d41dbdacc341ee93c86f7a7f056123" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546" is_pre_election: true
I20240503 13:56:37.712385 15758 raft_consensus.cc:2454] T 25d41dbdacc341ee93c86f7a7f056123 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 0.
I20240503 13:56:37.712390 15493 raft_consensus.cc:2454] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 0.
I20240503 13:56:37.712710 15825 leader_election.cc:302] T 25d41dbdacc341ee93c86f7a7f056123 P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61b887f0d40d497badeca5604c1f0546, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:37.712867 19032 raft_consensus.cc:2781] T 25d41dbdacc341ee93c86f7a7f056123 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:37.712915 19032 raft_consensus.cc:482] T 25d41dbdacc341ee93c86f7a7f056123 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:37.712950 19032 raft_consensus.cc:3037] T 25d41dbdacc341ee93c86f7a7f056123 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:37.713961 19032 raft_consensus.cc:504] T 25d41dbdacc341ee93c86f7a7f056123 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:37.714273 19032 leader_election.cc:288] T 25d41dbdacc341ee93c86f7a7f056123 P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 election: Requested vote from peers 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153), 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885)
I20240503 13:56:37.714591 15867 tablet_service.cc:1467] Processing CreateTablet for tablet a38035745c28423481d162ed67687215 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d5df17a465ba482794d26b2d3c9e1136]), partition=
I20240503 13:56:37.714841 15867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a38035745c28423481d162ed67687215. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:37.714869 15758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25d41dbdacc341ee93c86f7a7f056123" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2"
I20240503 13:56:37.714980 15605 tablet_service.cc:1467] Processing CreateTablet for tablet a38035745c28423481d162ed67687215 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d5df17a465ba482794d26b2d3c9e1136]), partition=
I20240503 13:56:37.714987 15736 tablet_service.cc:1467] Processing CreateTablet for tablet a38035745c28423481d162ed67687215 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d5df17a465ba482794d26b2d3c9e1136]), partition=
I20240503 13:56:37.715206 15736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a38035745c28423481d162ed67687215. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:37.715210 15605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a38035745c28423481d162ed67687215. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:37.716735 18755 tablet_bootstrap.cc:492] T a38035745c28423481d162ed67687215 P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:37.717041 15493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25d41dbdacc341ee93c86f7a7f056123" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546"
I20240503 13:56:37.714998 15758 raft_consensus.cc:3037] T 25d41dbdacc341ee93c86f7a7f056123 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:37.717182 15493 raft_consensus.cc:3037] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:37.717473 19121 tablet_bootstrap.cc:492] T a38035745c28423481d162ed67687215 P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:37.717622 18755 tablet_bootstrap.cc:654] T a38035745c28423481d162ed67687215 P e47105a0f46e4511b7112e571688a4c7: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:37.717999 15758 raft_consensus.cc:2454] T 25d41dbdacc341ee93c86f7a7f056123 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 1.
I20240503 13:56:37.718246 19121 tablet_bootstrap.cc:654] T a38035745c28423481d162ed67687215 P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:37.718243 15493 raft_consensus.cc:2454] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 1.
I20240503 13:56:37.718446 15825 leader_election.cc:302] T 25d41dbdacc341ee93c86f7a7f056123 P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76c386bef61a4bd5a907266a5118ffe2, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:37.718617 19032 raft_consensus.cc:2781] T 25d41dbdacc341ee93c86f7a7f056123 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:37.718726 19032 raft_consensus.cc:686] T 25d41dbdacc341ee93c86f7a7f056123 P e47105a0f46e4511b7112e571688a4c7 [term 1 LEADER]: Becoming Leader. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Running, Role: LEADER
I20240503 13:56:37.718787 18761 tablet_bootstrap.cc:492] T a38035745c28423481d162ed67687215 P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:37.718868 19032 consensus_queue.cc:237] T 25d41dbdacc341ee93c86f7a7f056123 P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:37.719400 18755 tablet_bootstrap.cc:492] T a38035745c28423481d162ed67687215 P e47105a0f46e4511b7112e571688a4c7: No bootstrap required, opened a new log
I20240503 13:56:37.719481 18755 ts_tablet_manager.cc:1367] T a38035745c28423481d162ed67687215 P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:56:37.719622 18761 tablet_bootstrap.cc:654] T a38035745c28423481d162ed67687215 P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:37.719887 18755 raft_consensus.cc:348] T a38035745c28423481d162ed67687215 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:37.719985 18755 raft_consensus.cc:374] T a38035745c28423481d162ed67687215 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:37.720014 18755 raft_consensus.cc:729] T a38035745c28423481d162ed67687215 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:37.720126 18755 consensus_queue.cc:260] T a38035745c28423481d162ed67687215 P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:37.720402 18755 ts_tablet_manager.cc:1398] T a38035745c28423481d162ed67687215 P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:37.721364 19121 tablet_bootstrap.cc:492] T a38035745c28423481d162ed67687215 P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:37.721452 19121 ts_tablet_manager.cc:1367] T a38035745c28423481d162ed67687215 P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:56:37.721612 12903 catalog_manager.cc:5484] T 25d41dbdacc341ee93c86f7a7f056123 P e47105a0f46e4511b7112e571688a4c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e47105a0f46e4511b7112e571688a4c7 (127.6.98.72). New cstate: current_term: 1 leader_uuid: "e47105a0f46e4511b7112e571688a4c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:37.721879 19121 raft_consensus.cc:348] T a38035745c28423481d162ed67687215 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:37.721997 19121 raft_consensus.cc:374] T a38035745c28423481d162ed67687215 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:37.722038 19121 raft_consensus.cc:729] T a38035745c28423481d162ed67687215 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:37.722152 19121 consensus_queue.cc:260] T a38035745c28423481d162ed67687215 P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:37.722414 19121 ts_tablet_manager.cc:1398] T a38035745c28423481d162ed67687215 P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:37.723531 18761 tablet_bootstrap.cc:492] T a38035745c28423481d162ed67687215 P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:37.723619 18761 ts_tablet_manager.cc:1367] T a38035745c28423481d162ed67687215 P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 13:56:37.724085 18761 raft_consensus.cc:348] T a38035745c28423481d162ed67687215 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:37.724182 18761 raft_consensus.cc:374] T a38035745c28423481d162ed67687215 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:37.724212 18761 raft_consensus.cc:729] T a38035745c28423481d162ed67687215 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:37.724323 18761 consensus_queue.cc:260] T a38035745c28423481d162ed67687215 P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:37.724635 18761 ts_tablet_manager.cc:1398] T a38035745c28423481d162ed67687215 P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:37.786994 19200 ts_tablet_manager.cc:902] T b82134ddae61479eb2ced348d7558799 P 76c386bef61a4bd5a907266a5118ffe2: Initiating tablet copy from peer 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:37.787556 19200 tablet_copy_client.cc:308] T b82134ddae61479eb2ced348d7558799 P 76c386bef61a4bd5a907266a5118ffe2: tablet copy: Beginning tablet copy session from remote peer at address 127.6.98.66:36387
I20240503 13:56:37.788146 13205 tablet_copy_service.cc:140] P 8ccf5a4ac450414c9d3579bd5800b770: Received BeginTabletCopySession request for tablet b82134ddae61479eb2ced348d7558799 from peer 76c386bef61a4bd5a907266a5118ffe2 ({username='slave'} at 127.6.98.71:41361)
I20240503 13:56:37.788268 13205 tablet_copy_service.cc:161] P 8ccf5a4ac450414c9d3579bd5800b770: Beginning new tablet copy session on tablet b82134ddae61479eb2ced348d7558799 from peer 76c386bef61a4bd5a907266a5118ffe2 at {username='slave'} at 127.6.98.71:41361: session id = 76c386bef61a4bd5a907266a5118ffe2-b82134ddae61479eb2ced348d7558799
I20240503 13:56:37.788573 18914 raft_consensus.cc:482] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:37.788686 18914 raft_consensus.cc:504] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:37.789319 13205 tablet_copy_source_session.cc:205] T b82134ddae61479eb2ced348d7558799 P 8ccf5a4ac450414c9d3579bd5800b770: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:56:37.789315 18914 leader_election.cc:288] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153)
I20240503 13:56:37.789835 15889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5a875bb0dce46f5883c302ea1729430" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7" is_pre_election: true
I20240503 13:56:37.789994 15889 raft_consensus.cc:2454] T a5a875bb0dce46f5883c302ea1729430 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 0.
I20240503 13:56:37.790303 13249 leader_election.cc:302] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba0194098aa84449ae17e8a0d4c43703, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:37.790518 18914 raft_consensus.cc:2781] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:37.790551 19200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b82134ddae61479eb2ced348d7558799. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:37.790588 18914 raft_consensus.cc:482] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:37.790628 18914 raft_consensus.cc:3037] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:37.791738 18914 raft_consensus.cc:504] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:37.792129 18914 leader_election.cc:288] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: Requested vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153)
I20240503 13:56:37.792467 15889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5a875bb0dce46f5883c302ea1729430" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7"
I20240503 13:56:37.792647 15889 raft_consensus.cc:3037] T a5a875bb0dce46f5883c302ea1729430 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:37.792802 19200 tablet_copy_client.cc:783] T b82134ddae61479eb2ced348d7558799 P 76c386bef61a4bd5a907266a5118ffe2: tablet copy: Starting download of 0 data blocks...
I20240503 13:56:37.792892 15493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5a875bb0dce46f5883c302ea1729430" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546" is_pre_election: true
I20240503 13:56:37.793007 19200 tablet_copy_client.cc:647] T b82134ddae61479eb2ced348d7558799 P 76c386bef61a4bd5a907266a5118ffe2: tablet copy: Starting download of 1 WAL segments...
I20240503 13:56:37.793032 15493 raft_consensus.cc:2454] T a5a875bb0dce46f5883c302ea1729430 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 0.
I20240503 13:56:37.793376 15493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5a875bb0dce46f5883c302ea1729430" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546"
I20240503 13:56:37.793481 15493 raft_consensus.cc:3037] T a5a875bb0dce46f5883c302ea1729430 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:37.793608 15889 raft_consensus.cc:2454] T a5a875bb0dce46f5883c302ea1729430 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 1.
I20240503 13:56:37.793896 13249 leader_election.cc:302] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba0194098aa84449ae17e8a0d4c43703, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:37.794068 18914 raft_consensus.cc:2781] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:37.794152 18914 raft_consensus.cc:686] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Becoming Leader. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Running, Role: LEADER
I20240503 13:56:37.794289 18914 consensus_queue.cc:237] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:37.794451 15493 raft_consensus.cc:2454] T a5a875bb0dce46f5883c302ea1729430 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 1.
I20240503 13:56:37.795812 19200 tablet_copy_client.cc:523] T b82134ddae61479eb2ced348d7558799 P 76c386bef61a4bd5a907266a5118ffe2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:56:37.796286 12907 catalog_manager.cc:5484] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 reported cstate change: term changed from 0 to 1, leader changed from <none> to ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67). New cstate: current_term: 1 leader_uuid: "ba0194098aa84449ae17e8a0d4c43703" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:37.797199 19200 tablet_bootstrap.cc:492] T b82134ddae61479eb2ced348d7558799 P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:37.818527 19200 tablet_bootstrap.cc:492] T b82134ddae61479eb2ced348d7558799 P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:56:37.818804 19200 tablet_bootstrap.cc:492] T b82134ddae61479eb2ced348d7558799 P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap complete.
I20240503 13:56:37.818989 19200 ts_tablet_manager.cc:1367] T b82134ddae61479eb2ced348d7558799 P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.022s user 0.020s sys 0.001s
I20240503 13:56:37.819434 19200 raft_consensus.cc:348] T b82134ddae61479eb2ced348d7558799 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:37.819561 19200 raft_consensus.cc:729] T b82134ddae61479eb2ced348d7558799 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:37.819697 19200 consensus_queue.cc:260] T b82134ddae61479eb2ced348d7558799 P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:37.820322 19200 ts_tablet_manager.cc:1398] T b82134ddae61479eb2ced348d7558799 P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:37.820902 13205 tablet_copy_service.cc:342] P 8ccf5a4ac450414c9d3579bd5800b770: Request end of tablet copy session 76c386bef61a4bd5a907266a5118ffe2-b82134ddae61479eb2ced348d7558799 received from {username='slave'} at 127.6.98.71:41361
I20240503 13:56:37.820986 13205 tablet_copy_service.cc:433] P 8ccf5a4ac450414c9d3579bd5800b770: ending tablet copy session 76c386bef61a4bd5a907266a5118ffe2-b82134ddae61479eb2ced348d7558799 on tablet b82134ddae61479eb2ced348d7558799 with peer 76c386bef61a4bd5a907266a5118ffe2
I20240503 13:56:37.829322 15472 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:56:37.834110 15736 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:56:37.836933 13164 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:37.841676 13424 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 13:56:37.841879 15605 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:37.852679 13295 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 13:56:37.864691 15867 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 13:56:37.867244 13034 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:37.885840 19180 ts_tablet_manager.cc:902] T a49b6f596d724b35b757ee69fe15e5a5 P 61b887f0d40d497badeca5604c1f0546: Initiating tablet copy from peer 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:37.886323 19180 tablet_copy_client.cc:308] T a49b6f596d724b35b757ee69fe15e5a5 P 61b887f0d40d497badeca5604c1f0546: tablet copy: Beginning tablet copy session from remote peer at address 127.6.98.66:36387
I20240503 13:56:37.886938 13205 tablet_copy_service.cc:140] P 8ccf5a4ac450414c9d3579bd5800b770: Received BeginTabletCopySession request for tablet a49b6f596d724b35b757ee69fe15e5a5 from peer 61b887f0d40d497badeca5604c1f0546 ({username='slave'} at 127.6.98.69:41857)
I20240503 13:56:37.887065 13205 tablet_copy_service.cc:161] P 8ccf5a4ac450414c9d3579bd5800b770: Beginning new tablet copy session on tablet a49b6f596d724b35b757ee69fe15e5a5 from peer 61b887f0d40d497badeca5604c1f0546 at {username='slave'} at 127.6.98.69:41857: session id = 61b887f0d40d497badeca5604c1f0546-a49b6f596d724b35b757ee69fe15e5a5
I20240503 13:56:37.888115 13205 tablet_copy_source_session.cc:205] T a49b6f596d724b35b757ee69fe15e5a5 P 8ccf5a4ac450414c9d3579bd5800b770: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:56:37.889098 19180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a49b6f596d724b35b757ee69fe15e5a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:37.891301 19180 tablet_copy_client.cc:783] T a49b6f596d724b35b757ee69fe15e5a5 P 61b887f0d40d497badeca5604c1f0546: tablet copy: Starting download of 0 data blocks...
I20240503 13:56:37.891448 19180 tablet_copy_client.cc:647] T a49b6f596d724b35b757ee69fe15e5a5 P 61b887f0d40d497badeca5604c1f0546: tablet copy: Starting download of 1 WAL segments...
I20240503 13:56:37.893779 19180 tablet_copy_client.cc:523] T a49b6f596d724b35b757ee69fe15e5a5 P 61b887f0d40d497badeca5604c1f0546: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:56:37.895089 19180 tablet_bootstrap.cc:492] T a49b6f596d724b35b757ee69fe15e5a5 P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:37.924913 19180 tablet_bootstrap.cc:492] T a49b6f596d724b35b757ee69fe15e5a5 P 61b887f0d40d497badeca5604c1f0546: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:56:37.925256 19180 tablet_bootstrap.cc:492] T a49b6f596d724b35b757ee69fe15e5a5 P 61b887f0d40d497badeca5604c1f0546: Bootstrap complete.
I20240503 13:56:37.925493 19180 ts_tablet_manager.cc:1367] T a49b6f596d724b35b757ee69fe15e5a5 P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.030s user 0.024s sys 0.002s
I20240503 13:56:37.925995 19180 raft_consensus.cc:348] T a49b6f596d724b35b757ee69fe15e5a5 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:37.926121 19180 raft_consensus.cc:729] T a49b6f596d724b35b757ee69fe15e5a5 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:37.926246 19180 consensus_queue.cc:260] T a49b6f596d724b35b757ee69fe15e5a5 P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:37.926548 19180 ts_tablet_manager.cc:1398] T a49b6f596d724b35b757ee69fe15e5a5 P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:37.927059 13205 tablet_copy_service.cc:342] P 8ccf5a4ac450414c9d3579bd5800b770: Request end of tablet copy session 61b887f0d40d497badeca5604c1f0546-a49b6f596d724b35b757ee69fe15e5a5 received from {username='slave'} at 127.6.98.69:41857
I20240503 13:56:37.927152 13205 tablet_copy_service.cc:433] P 8ccf5a4ac450414c9d3579bd5800b770: ending tablet copy session 61b887f0d40d497badeca5604c1f0546-a49b6f596d724b35b757ee69fe15e5a5 on tablet a49b6f596d724b35b757ee69fe15e5a5 with peer 61b887f0d40d497badeca5604c1f0546
I20240503 13:56:37.929916 18752 raft_consensus.cc:482] T a38035745c28423481d162ed67687215 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:37.930034 18752 raft_consensus.cc:504] T a38035745c28423481d162ed67687215 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:37.930404 18752 leader_election.cc:288] T a38035745c28423481d162ed67687215 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:37.931207 15889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38035745c28423481d162ed67687215" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7" is_pre_election: true
I20240503 13:56:37.931351 15889 raft_consensus.cc:2454] T a38035745c28423481d162ed67687215 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 0.
I20240503 13:56:37.931903 15625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38035745c28423481d162ed67687215" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d" is_pre_election: true
I20240503 13:56:37.932030 15625 raft_consensus.cc:2454] T a38035745c28423481d162ed67687215 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 0.
I20240503 13:56:37.932264 15691 leader_election.cc:302] T a38035745c28423481d162ed67687215 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76c386bef61a4bd5a907266a5118ffe2, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:37.932587 18752 raft_consensus.cc:2781] T a38035745c28423481d162ed67687215 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:37.932650 18752 raft_consensus.cc:482] T a38035745c28423481d162ed67687215 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:37.932683 18752 raft_consensus.cc:3037] T a38035745c28423481d162ed67687215 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:37.933746 18752 raft_consensus.cc:504] T a38035745c28423481d162ed67687215 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:37.934115 18752 leader_election.cc:288] T a38035745c28423481d162ed67687215 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 election: Requested vote from peers 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:37.934437 15889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38035745c28423481d162ed67687215" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7"
I20240503 13:56:37.934562 15889 raft_consensus.cc:3037] T a38035745c28423481d162ed67687215 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:37.935487 15889 raft_consensus.cc:2454] T a38035745c28423481d162ed67687215 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 1.
I20240503 13:56:37.935806 15691 leader_election.cc:302] T a38035745c28423481d162ed67687215 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76c386bef61a4bd5a907266a5118ffe2, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:37.935956 18752 raft_consensus.cc:2781] T a38035745c28423481d162ed67687215 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:37.935985 15625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38035745c28423481d162ed67687215" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d"
I20240503 13:56:37.936045 18752 raft_consensus.cc:686] T a38035745c28423481d162ed67687215 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 LEADER]: Becoming Leader. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Running, Role: LEADER
I20240503 13:56:37.936098 15625 raft_consensus.cc:3037] T a38035745c28423481d162ed67687215 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:37.936172 18752 consensus_queue.cc:237] T a38035745c28423481d162ed67687215 P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:37.937000 15625 raft_consensus.cc:2454] T a38035745c28423481d162ed67687215 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 1.
I20240503 13:56:37.937951 12903 catalog_manager.cc:5484] T a38035745c28423481d162ed67687215 P 76c386bef61a4bd5a907266a5118ffe2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71). New cstate: current_term: 1 leader_uuid: "76c386bef61a4bd5a907266a5118ffe2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:37.967545 18752 consensus_queue.cc:1035] T e9849f54796d477db114fa7a85257fe2 P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:37.975062 18645 consensus_queue.cc:1035] T e9849f54796d477db114fa7a85257fe2 P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:38.007757 15605 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:38.010002 13034 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:38.019400 13164 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:38.020655 15736 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:56:38.022439 15472 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:56:38.024998 13424 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 13:56:38.042182 13295 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 13:56:38.046659 15867 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 13:56:38.082204 12903 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:56:38.086423 15889 raft_consensus.cc:1261] T a38035745c28423481d162ed67687215 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Refusing update from remote peer 76c386bef61a4bd5a907266a5118ffe2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:38.086581 15625 raft_consensus.cc:1261] T a38035745c28423481d162ed67687215 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Refusing update from remote peer 76c386bef61a4bd5a907266a5118ffe2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:38.087059 18752 consensus_queue.cc:1035] T a38035745c28423481d162ed67687215 P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:38.087384 18645 consensus_queue.cc:1035] T a38035745c28423481d162ed67687215 P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:38.089963 19304 tablet.cc:1722] T a38035745c28423481d162ed67687215 P 76c386bef61a4bd5a907266a5118ffe2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:38.090363 19304 tablet.cc:1620] T a38035745c28423481d162ed67687215 P 76c386bef61a4bd5a907266a5118ffe2: MemRowSet was empty: no flush needed.
I20240503 13:56:38.090494 15889 raft_consensus.cc:1203] T d50d05d432e84d5eb0e54d1a68cdf158 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:56:38.091231 12909 catalog_manager.cc:5576] TS e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753) does not have the latest schema for tablet a38035745c28423481d162ed67687215 (table rebalancer_dynamic_table [id=d5df17a465ba482794d26b2d3c9e1136]). Expected version 1 got 0
I20240503 13:56:38.093127 19305 tablet.cc:1722] T a38035745c28423481d162ed67687215 P 009f849ee7b54c80a3f5264695047e4d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:38.093292 12905 catalog_manager.cc:5576] TS 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951) does not have the latest schema for tablet a38035745c28423481d162ed67687215 (table rebalancer_dynamic_table [id=d5df17a465ba482794d26b2d3c9e1136]). Expected version 1 got 0
I20240503 13:56:38.093761 19305 tablet.cc:1620] T a38035745c28423481d162ed67687215 P 009f849ee7b54c80a3f5264695047e4d: MemRowSet was empty: no flush needed.
I20240503 13:56:38.097990 12909 catalog_manager.cc:5576] TS e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753) does not have the latest schema for tablet a38035745c28423481d162ed67687215 (table rebalancer_dynamic_table [id=d5df17a465ba482794d26b2d3c9e1136]). Expected version 1 got 0
I20240503 13:56:38.100641 19306 tablet.cc:1722] T a38035745c28423481d162ed67687215 P e47105a0f46e4511b7112e571688a4c7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:38.101063 19306 tablet.cc:1620] T a38035745c28423481d162ed67687215 P e47105a0f46e4511b7112e571688a4c7: MemRowSet was empty: no flush needed.
I20240503 13:56:38.101063 12905 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=d5df17a465ba482794d26b2d3c9e1136] alter complete (version 1)
I20240503 13:56:38.118820 13314 raft_consensus.cc:1203] T 0878ccb6135f4485b5ba35ae67a76d0c P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:56:38.161695 13445 raft_consensus.cc:1203] T 27d5ced54d2e4aa9b6ea01fe8700443d P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:56:38.174844 15867 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 13:56:38.177134 15736 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:56:38.180016 15472 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:56:38.189458 15605 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:38.193662 13424 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 13:56:38.193799 13034 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:38.202622 15493 raft_consensus.cc:1203] T 38596c626067458495846931e0ecb7eb P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:56:38.202708 13164 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:38.215449 12903 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:56:38.220922 17953 consensus_queue.cc:1035] T 25d41dbdacc341ee93c86f7a7f056123 P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:38.223829 13295 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 13:56:38.223984 12904 catalog_manager.cc:5576] TS 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885) does not have the latest schema for tablet a38035745c28423481d162ed67687215 (table rebalancer_dynamic_table [id=d5df17a465ba482794d26b2d3c9e1136]). Expected version 2 got 1
I20240503 13:56:38.228194 19304 tablet.cc:1722] T a38035745c28423481d162ed67687215 P 76c386bef61a4bd5a907266a5118ffe2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:38.228539 19304 tablet.cc:1620] T a38035745c28423481d162ed67687215 P 76c386bef61a4bd5a907266a5118ffe2: MemRowSet was empty: no flush needed.
I20240503 13:56:38.229633 15889 raft_consensus.cc:1203] T 477c33f203f649019eff2b28524fad90 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:56:38.232309 19306 tablet.cc:1722] T a38035745c28423481d162ed67687215 P e47105a0f46e4511b7112e571688a4c7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:38.232771 19306 tablet.cc:1620] T a38035745c28423481d162ed67687215 P e47105a0f46e4511b7112e571688a4c7: MemRowSet was empty: no flush needed.
I20240503 13:56:38.234117 19305 tablet.cc:1722] T a38035745c28423481d162ed67687215 P 009f849ee7b54c80a3f5264695047e4d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:38.234409 19305 tablet.cc:1620] T a38035745c28423481d162ed67687215 P 009f849ee7b54c80a3f5264695047e4d: MemRowSet was empty: no flush needed.
I20240503 13:56:38.238711 15757 raft_consensus.cc:1203] T dc544319be0b484f8e9e5a3fbc40f530 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:56:38.240239 12912 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=d5df17a465ba482794d26b2d3c9e1136] alter complete (version 2)
I20240503 13:56:38.258209 17953 consensus_queue.cc:1035] T 25d41dbdacc341ee93c86f7a7f056123 P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:38.314162 15758 raft_consensus.cc:1203] T b82134ddae61479eb2ced348d7558799 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:56:38.322042 12912 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_extra_table_24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:56:38.329186 15472 tablet_service.cc:1467] Processing CreateTablet for tablet cdc199537531406ab832e88682140993 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=2e7d0ad3cc8e4e18b0ac718e44b319a8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:38.329372 15736 tablet_service.cc:1467] Processing CreateTablet for tablet cdc199537531406ab832e88682140993 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=2e7d0ad3cc8e4e18b0ac718e44b319a8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:38.329535 15472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdc199537531406ab832e88682140993. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:38.329666 15736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdc199537531406ab832e88682140993. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:38.329799 13164 tablet_service.cc:1467] Processing CreateTablet for tablet cdc199537531406ab832e88682140993 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=2e7d0ad3cc8e4e18b0ac718e44b319a8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:38.330045 13164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdc199537531406ab832e88682140993. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:38.332851 19396 tablet_bootstrap.cc:492] T cdc199537531406ab832e88682140993 P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:38.333076 15471 tablet_service.cc:1467] Processing CreateTablet for tablet bf4ea1072ad542d89fda48ca4e839c79 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=2e7d0ad3cc8e4e18b0ac718e44b319a8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:38.333317 15471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf4ea1072ad542d89fda48ca4e839c79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:38.333374 13164 tablet_service.cc:1467] Processing CreateTablet for tablet 3e30d88e2c054de6b87e4beb5eb14019 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=2e7d0ad3cc8e4e18b0ac718e44b319a8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:38.333606 13164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e30d88e2c054de6b87e4beb5eb14019. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:38.333863 19396 tablet_bootstrap.cc:654] T cdc199537531406ab832e88682140993 P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:38.334941 19396 tablet_bootstrap.cc:492] T cdc199537531406ab832e88682140993 P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:38.335028 19396 ts_tablet_manager.cc:1367] T cdc199537531406ab832e88682140993 P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:56:38.335736 19396 raft_consensus.cc:348] T cdc199537531406ab832e88682140993 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:38.336282 15867 tablet_service.cc:1467] Processing CreateTablet for tablet 3e30d88e2c054de6b87e4beb5eb14019 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=2e7d0ad3cc8e4e18b0ac718e44b319a8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:38.336611 15867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e30d88e2c054de6b87e4beb5eb14019. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:38.336771 15605 tablet_service.cc:1467] Processing CreateTablet for tablet bf4ea1072ad542d89fda48ca4e839c79 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=2e7d0ad3cc8e4e18b0ac718e44b319a8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:38.337036 15605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf4ea1072ad542d89fda48ca4e839c79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:38.337210 15868 tablet_service.cc:1467] Processing CreateTablet for tablet bf4ea1072ad542d89fda48ca4e839c79 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=2e7d0ad3cc8e4e18b0ac718e44b319a8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:38.337425 15868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf4ea1072ad542d89fda48ca4e839c79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:38.337754 19395 tablet_bootstrap.cc:492] T cdc199537531406ab832e88682140993 P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:38.338778 19395 tablet_bootstrap.cc:654] T cdc199537531406ab832e88682140993 P 61b887f0d40d497badeca5604c1f0546: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:38.339408 19397 tablet_bootstrap.cc:492] T cdc199537531406ab832e88682140993 P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:38.340075 15472 tablet_service.cc:1467] Processing CreateTablet for tablet 3e30d88e2c054de6b87e4beb5eb14019 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=2e7d0ad3cc8e4e18b0ac718e44b319a8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:38.340310 15472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e30d88e2c054de6b87e4beb5eb14019. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:38.340457 19397 tablet_bootstrap.cc:654] T cdc199537531406ab832e88682140993 P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:38.342679 12911 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:56:38.345616 19395 tablet_bootstrap.cc:492] T cdc199537531406ab832e88682140993 P 61b887f0d40d497badeca5604c1f0546: No bootstrap required, opened a new log
I20240503 13:56:38.346856 19395 ts_tablet_manager.cc:1367] T cdc199537531406ab832e88682140993 P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 13:56:38.349342 19396 raft_consensus.cc:374] T cdc199537531406ab832e88682140993 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:38.349467 19396 raft_consensus.cc:729] T cdc199537531406ab832e88682140993 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:38.348831 19395 raft_consensus.cc:348] T cdc199537531406ab832e88682140993 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:38.349678 19396 consensus_queue.cc:260] T cdc199537531406ab832e88682140993 P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:38.349880 19395 raft_consensus.cc:374] T cdc199537531406ab832e88682140993 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:38.349962 19395 raft_consensus.cc:729] T cdc199537531406ab832e88682140993 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:38.350076 19396 ts_tablet_manager.cc:1398] T cdc199537531406ab832e88682140993 P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.015s user 0.001s sys 0.000s
I20240503 13:56:38.350198 19395 consensus_queue.cc:260] T cdc199537531406ab832e88682140993 P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:38.350359 19400 tablet_bootstrap.cc:492] T bf4ea1072ad542d89fda48ca4e839c79 P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:38.351850 19304 tablet.cc:1722] T a38035745c28423481d162ed67687215 P 76c386bef61a4bd5a907266a5118ffe2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:38.355190 19305 tablet.cc:1722] T a38035745c28423481d162ed67687215 P 009f849ee7b54c80a3f5264695047e4d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:38.355787 19395 ts_tablet_manager.cc:1398] T cdc199537531406ab832e88682140993 P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20240503 13:56:38.356142 19306 tablet.cc:1722] T a38035745c28423481d162ed67687215 P e47105a0f46e4511b7112e571688a4c7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:38.356801 19395 tablet_bootstrap.cc:492] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:38.358165 19395 tablet_bootstrap.cc:654] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:38.362773 12908 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table_new_name [id=d5df17a465ba482794d26b2d3c9e1136] alter complete (version 3)
I20240503 13:56:38.362847 18873 consensus_queue.cc:1035] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:38.371160 19400 tablet_bootstrap.cc:654] T bf4ea1072ad542d89fda48ca4e839c79 P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:38.371407 19399 tablet_bootstrap.cc:492] T 3e30d88e2c054de6b87e4beb5eb14019 P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:38.372545 18873 consensus_queue.cc:1035] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:38.373371 19399 tablet_bootstrap.cc:654] T 3e30d88e2c054de6b87e4beb5eb14019 P e47105a0f46e4511b7112e571688a4c7: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:38.376448 19397 tablet_bootstrap.cc:492] T cdc199537531406ab832e88682140993 P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:38.376608 19397 ts_tablet_manager.cc:1367] T cdc199537531406ab832e88682140993 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.037s user 0.002s sys 0.000s
I20240503 13:56:38.377091 19397 raft_consensus.cc:348] T cdc199537531406ab832e88682140993 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:38.377197 19397 raft_consensus.cc:374] T cdc199537531406ab832e88682140993 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:38.377228 19397 raft_consensus.cc:729] T cdc199537531406ab832e88682140993 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:38.377344 19397 consensus_queue.cc:260] T cdc199537531406ab832e88682140993 P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:38.377645 19397 ts_tablet_manager.cc:1398] T cdc199537531406ab832e88682140993 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:38.377825 19397 tablet_bootstrap.cc:492] T 3e30d88e2c054de6b87e4beb5eb14019 P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:38.378728 19397 tablet_bootstrap.cc:654] T 3e30d88e2c054de6b87e4beb5eb14019 P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:38.382349 19399 tablet_bootstrap.cc:492] T 3e30d88e2c054de6b87e4beb5eb14019 P e47105a0f46e4511b7112e571688a4c7: No bootstrap required, opened a new log
I20240503 13:56:38.382553 19399 ts_tablet_manager.cc:1367] T 3e30d88e2c054de6b87e4beb5eb14019 P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.002s
I20240503 13:56:38.383042 19399 raft_consensus.cc:348] T 3e30d88e2c054de6b87e4beb5eb14019 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:38.383154 19399 raft_consensus.cc:374] T 3e30d88e2c054de6b87e4beb5eb14019 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:38.383186 19399 raft_consensus.cc:729] T 3e30d88e2c054de6b87e4beb5eb14019 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:38.383307 19399 consensus_queue.cc:260] T 3e30d88e2c054de6b87e4beb5eb14019 P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:38.383687 19399 ts_tablet_manager.cc:1398] T 3e30d88e2c054de6b87e4beb5eb14019 P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:38.383880 19399 tablet_bootstrap.cc:492] T bf4ea1072ad542d89fda48ca4e839c79 P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:38.386751 19399 tablet_bootstrap.cc:654] T bf4ea1072ad542d89fda48ca4e839c79 P e47105a0f46e4511b7112e571688a4c7: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:38.389969 19399 tablet_bootstrap.cc:492] T bf4ea1072ad542d89fda48ca4e839c79 P e47105a0f46e4511b7112e571688a4c7: No bootstrap required, opened a new log
I20240503 13:56:38.390617 19399 ts_tablet_manager.cc:1367] T bf4ea1072ad542d89fda48ca4e839c79 P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 13:56:38.391691 19399 raft_consensus.cc:348] T bf4ea1072ad542d89fda48ca4e839c79 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:38.392055 19399 raft_consensus.cc:374] T bf4ea1072ad542d89fda48ca4e839c79 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:38.392143 19399 raft_consensus.cc:729] T bf4ea1072ad542d89fda48ca4e839c79 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:38.392145 15605 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:38.392462 19399 consensus_queue.cc:260] T bf4ea1072ad542d89fda48ca4e839c79 P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:38.392947 19399 ts_tablet_manager.cc:1398] T bf4ea1072ad542d89fda48ca4e839c79 P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:38.393837 19397 tablet_bootstrap.cc:492] T 3e30d88e2c054de6b87e4beb5eb14019 P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:38.393930 19397 ts_tablet_manager.cc:1367] T 3e30d88e2c054de6b87e4beb5eb14019 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240503 13:56:38.394371 19397 raft_consensus.cc:348] T 3e30d88e2c054de6b87e4beb5eb14019 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:38.394466 19397 raft_consensus.cc:374] T 3e30d88e2c054de6b87e4beb5eb14019 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:38.394493 19397 raft_consensus.cc:729] T 3e30d88e2c054de6b87e4beb5eb14019 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:38.394603 19397 consensus_queue.cc:260] T 3e30d88e2c054de6b87e4beb5eb14019 P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:38.394857 19397 ts_tablet_manager.cc:1398] T 3e30d88e2c054de6b87e4beb5eb14019 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:38.396564 15472 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:56:38.397413 19400 tablet_bootstrap.cc:492] T bf4ea1072ad542d89fda48ca4e839c79 P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:38.397502 19400 ts_tablet_manager.cc:1367] T bf4ea1072ad542d89fda48ca4e839c79 P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.047s user 0.002s sys 0.000s
I20240503 13:56:38.397913 19400 raft_consensus.cc:348] T bf4ea1072ad542d89fda48ca4e839c79 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:38.398016 19400 raft_consensus.cc:374] T bf4ea1072ad542d89fda48ca4e839c79 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:38.398041 19400 raft_consensus.cc:729] T bf4ea1072ad542d89fda48ca4e839c79 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:38.398140 19400 consensus_queue.cc:260] T bf4ea1072ad542d89fda48ca4e839c79 P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:38.398370 19400 ts_tablet_manager.cc:1398] T bf4ea1072ad542d89fda48ca4e839c79 P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:38.407752 13034 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:38.409551 19395 tablet_bootstrap.cc:492] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546: No bootstrap required, opened a new log
I20240503 13:56:38.409703 19395 ts_tablet_manager.cc:1367] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.053s user 0.002s sys 0.000s
I20240503 13:56:38.410223 19395 raft_consensus.cc:348] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:38.410359 19395 raft_consensus.cc:374] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:38.410399 19395 raft_consensus.cc:729] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:38.410519 19395 consensus_queue.cc:260] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:38.410825 19395 ts_tablet_manager.cc:1398] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 13:56:38.411051 19395 tablet_bootstrap.cc:492] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:38.412070 19395 tablet_bootstrap.cc:654] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:38.424022 13424 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 13:56:38.427505 13164 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:38.427742 13295 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 13:56:38.431254 19395 tablet_bootstrap.cc:492] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546: No bootstrap required, opened a new log
I20240503 13:56:38.431361 19395 ts_tablet_manager.cc:1367] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20240503 13:56:38.431809 19395 raft_consensus.cc:348] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:38.431932 19395 raft_consensus.cc:374] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:38.431968 19395 raft_consensus.cc:729] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:38.432085 19395 consensus_queue.cc:260] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:38.432343 19395 ts_tablet_manager.cc:1398] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:38.436805 15868 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 13:56:38.439074 15736 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:56:38.441399 15493 raft_consensus.cc:1203] T a49b6f596d724b35b757ee69fe15e5a5 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:56:38.449784 19349 raft_consensus.cc:482] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:38.449899 19349 raft_consensus.cc:504] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:38.450269 19349 leader_election.cc:288] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:38.450649 15888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf4ea1072ad542d89fda48ca4e839c79" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7" is_pre_election: true
I20240503 13:56:38.450729 15625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf4ea1072ad542d89fda48ca4e839c79" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d" is_pre_election: true
I20240503 13:56:38.450785 15888 raft_consensus.cc:2454] T bf4ea1072ad542d89fda48ca4e839c79 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 0.
I20240503 13:56:38.450853 15625 raft_consensus.cc:2454] T bf4ea1072ad542d89fda48ca4e839c79 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 0.
I20240503 13:56:38.451126 15426 leader_election.cc:302] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 61b887f0d40d497badeca5604c1f0546; no voters:
I20240503 13:56:38.451364 19349 raft_consensus.cc:2781] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:38.451411 19349 raft_consensus.cc:482] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:38.451448 19349 raft_consensus.cc:3037] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:38.452575 19349 raft_consensus.cc:504] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:38.452929 19349 leader_election.cc:288] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 election: Requested vote from peers 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:38.453238 15625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf4ea1072ad542d89fda48ca4e839c79" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d"
I20240503 13:56:38.453346 15625 raft_consensus.cc:3037] T bf4ea1072ad542d89fda48ca4e839c79 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:38.453287 15888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf4ea1072ad542d89fda48ca4e839c79" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7"
I20240503 13:56:38.453420 15888 raft_consensus.cc:3037] T bf4ea1072ad542d89fda48ca4e839c79 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:38.454274 15625 raft_consensus.cc:2454] T bf4ea1072ad542d89fda48ca4e839c79 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 1.
I20240503 13:56:38.454367 15888 raft_consensus.cc:2454] T bf4ea1072ad542d89fda48ca4e839c79 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 1.
I20240503 13:56:38.454622 15426 leader_election.cc:302] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 61b887f0d40d497badeca5604c1f0546; no voters:
I20240503 13:56:38.454859 19349 raft_consensus.cc:2781] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:38.455010 19349 raft_consensus.cc:686] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 [term 1 LEADER]: Becoming Leader. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Running, Role: LEADER
I20240503 13:56:38.455138 19349 consensus_queue.cc:237] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:38.459143 12908 catalog_manager.cc:5484] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61b887f0d40d497badeca5604c1f0546 (127.6.98.69). New cstate: current_term: 1 leader_uuid: "61b887f0d40d497badeca5604c1f0546" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:38.475538 12908 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:38.475730 12908 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:38.477909 12908 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Sending DeleteTablet for 3 replicas of tablet a38035745c28423481d162ed67687215
I20240503 13:56:38.478451 15605 tablet_service.cc:1514] Processing DeleteTablet for tablet a38035745c28423481d162ed67687215 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:38 UTC) from {username='slave'} at 127.0.0.1:60718
I20240503 13:56:38.478602 15736 tablet_service.cc:1514] Processing DeleteTablet for tablet a38035745c28423481d162ed67687215 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:38 UTC) from {username='slave'} at 127.0.0.1:44984
I20240503 13:56:38.478830 15868 tablet_service.cc:1514] Processing DeleteTablet for tablet a38035745c28423481d162ed67687215 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:38 UTC) from {username='slave'} at 127.0.0.1:50498
I20240503 13:56:38.478891 19427 tablet_replica.cc:330] T a38035745c28423481d162ed67687215 P 76c386bef61a4bd5a907266a5118ffe2: stopping tablet replica
I20240503 13:56:38.479043 19427 raft_consensus.cc:2229] T a38035745c28423481d162ed67687215 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:56:38.479197 19426 tablet_replica.cc:330] T a38035745c28423481d162ed67687215 P 009f849ee7b54c80a3f5264695047e4d: stopping tablet replica
I20240503 13:56:38.479269 19427 raft_consensus.cc:2258] T a38035745c28423481d162ed67687215 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:38.479349 19426 raft_consensus.cc:2229] T a38035745c28423481d162ed67687215 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:38.479476 19426 raft_consensus.cc:2258] T a38035745c28423481d162ed67687215 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:38.480000 19427 ts_tablet_manager.cc:1875] T a38035745c28423481d162ed67687215 P 76c386bef61a4bd5a907266a5118ffe2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:38.480049 19426 ts_tablet_manager.cc:1875] T a38035745c28423481d162ed67687215 P 009f849ee7b54c80a3f5264695047e4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:38.482268 19427 ts_tablet_manager.cc:1888] T a38035745c28423481d162ed67687215 P 76c386bef61a4bd5a907266a5118ffe2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:38.482372 19427 log.cc:1196] T a38035745c28423481d162ed67687215 P 76c386bef61a4bd5a907266a5118ffe2: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/wal/wals/a38035745c28423481d162ed67687215
I20240503 13:56:38.482646 19427 ts_tablet_manager.cc:1909] T a38035745c28423481d162ed67687215 P 76c386bef61a4bd5a907266a5118ffe2: Deleting consensus metadata
I20240503 13:56:38.483534 12899 catalog_manager.cc:4830] TS 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885): tablet a38035745c28423481d162ed67687215 (table rebalancer_dynamic_table_new_name [id=d5df17a465ba482794d26b2d3c9e1136]) successfully deleted
I20240503 13:56:38.484563 19426 ts_tablet_manager.cc:1888] T a38035745c28423481d162ed67687215 P 009f849ee7b54c80a3f5264695047e4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:38.484733 19426 log.cc:1196] T a38035745c28423481d162ed67687215 P 009f849ee7b54c80a3f5264695047e4d: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/wal/wals/a38035745c28423481d162ed67687215
I20240503 13:56:38.485029 19428 tablet_replica.cc:330] T a38035745c28423481d162ed67687215 P e47105a0f46e4511b7112e571688a4c7: stopping tablet replica
I20240503 13:56:38.485072 19426 ts_tablet_manager.cc:1909] T a38035745c28423481d162ed67687215 P 009f849ee7b54c80a3f5264695047e4d: Deleting consensus metadata
I20240503 13:56:38.485186 19428 raft_consensus.cc:2229] T a38035745c28423481d162ed67687215 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:38.485384 19428 raft_consensus.cc:2258] T a38035745c28423481d162ed67687215 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:38.486033 12896 catalog_manager.cc:4830] TS 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951): tablet a38035745c28423481d162ed67687215 (table rebalancer_dynamic_table_new_name [id=d5df17a465ba482794d26b2d3c9e1136]) successfully deleted
I20240503 13:56:38.486820 19428 ts_tablet_manager.cc:1875] T a38035745c28423481d162ed67687215 P e47105a0f46e4511b7112e571688a4c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:38.488993 19428 ts_tablet_manager.cc:1888] T a38035745c28423481d162ed67687215 P e47105a0f46e4511b7112e571688a4c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:38.489099 19428 log.cc:1196] T a38035745c28423481d162ed67687215 P e47105a0f46e4511b7112e571688a4c7: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-7/wal/wals/a38035745c28423481d162ed67687215
I20240503 13:56:38.489362 19428 ts_tablet_manager.cc:1909] T a38035745c28423481d162ed67687215 P e47105a0f46e4511b7112e571688a4c7: Deleting consensus metadata
I20240503 13:56:38.490155 12896 catalog_manager.cc:4830] TS e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): tablet a38035745c28423481d162ed67687215 (table rebalancer_dynamic_table_new_name [id=d5df17a465ba482794d26b2d3c9e1136]) successfully deleted
I20240503 13:56:38.498003 19349 raft_consensus.cc:482] T cdc199537531406ab832e88682140993 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:38.498133 19349 raft_consensus.cc:504] T cdc199537531406ab832e88682140993 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:38.498561 19349 leader_election.cc:288] T cdc199537531406ab832e88682140993 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387), 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885)
I20240503 13:56:38.498865 13185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc199537531406ab832e88682140993" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770" is_pre_election: true
I20240503 13:56:38.498998 13185 raft_consensus.cc:2454] T cdc199537531406ab832e88682140993 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 0.
I20240503 13:56:38.498983 15758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc199537531406ab832e88682140993" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2" is_pre_election: true
I20240503 13:56:38.499123 15758 raft_consensus.cc:2454] T cdc199537531406ab832e88682140993 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 0.
I20240503 13:56:38.499289 15429 leader_election.cc:302] T cdc199537531406ab832e88682140993 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61b887f0d40d497badeca5604c1f0546, 8ccf5a4ac450414c9d3579bd5800b770; no voters:
I20240503 13:56:38.499537 19349 raft_consensus.cc:2781] T cdc199537531406ab832e88682140993 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:38.499584 19349 raft_consensus.cc:482] T cdc199537531406ab832e88682140993 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:38.499619 19349 raft_consensus.cc:3037] T cdc199537531406ab832e88682140993 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:38.500667 19349 raft_consensus.cc:504] T cdc199537531406ab832e88682140993 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:38.500964 19349 leader_election.cc:288] T cdc199537531406ab832e88682140993 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 election: Requested vote from peers 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387), 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885)
I20240503 13:56:38.501588 13185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc199537531406ab832e88682140993" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770"
I20240503 13:56:38.501626 15758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc199537531406ab832e88682140993" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2"
I20240503 13:56:38.501698 13185 raft_consensus.cc:3037] T cdc199537531406ab832e88682140993 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:38.501760 15758 raft_consensus.cc:3037] T cdc199537531406ab832e88682140993 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:38.502794 15758 raft_consensus.cc:2454] T cdc199537531406ab832e88682140993 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 1.
I20240503 13:56:38.502846 13185 raft_consensus.cc:2454] T cdc199537531406ab832e88682140993 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 1.
I20240503 13:56:38.503096 15429 leader_election.cc:302] T cdc199537531406ab832e88682140993 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61b887f0d40d497badeca5604c1f0546, 76c386bef61a4bd5a907266a5118ffe2; no voters:
I20240503 13:56:38.503412 19349 raft_consensus.cc:2781] T cdc199537531406ab832e88682140993 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:38.503496 19349 raft_consensus.cc:686] T cdc199537531406ab832e88682140993 P 61b887f0d40d497badeca5604c1f0546 [term 1 LEADER]: Becoming Leader. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Running, Role: LEADER
I20240503 13:56:38.503628 19349 consensus_queue.cc:237] T cdc199537531406ab832e88682140993 P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:38.505198 19445 raft_consensus.cc:482] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:38.505331 19445 raft_consensus.cc:504] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:38.505718 19445 leader_election.cc:288] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:38.506038 15888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e30d88e2c054de6b87e4beb5eb14019" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7" is_pre_election: true
I20240503 13:56:38.506168 15888 raft_consensus.cc:2454] T 3e30d88e2c054de6b87e4beb5eb14019 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 0.
I20240503 13:56:38.506131 12908 catalog_manager.cc:5484] T cdc199537531406ab832e88682140993 P 61b887f0d40d497badeca5604c1f0546 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61b887f0d40d497badeca5604c1f0546 (127.6.98.69). New cstate: current_term: 1 leader_uuid: "61b887f0d40d497badeca5604c1f0546" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:38.506395 13185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e30d88e2c054de6b87e4beb5eb14019" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770" is_pre_election: true
I20240503 13:56:38.506458 15426 leader_election.cc:302] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61b887f0d40d497badeca5604c1f0546, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:38.506541 13185 raft_consensus.cc:2454] T 3e30d88e2c054de6b87e4beb5eb14019 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 0.
I20240503 13:56:38.507256 15492 consensus_queue.cc:237] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:38.508003 19349 raft_consensus.cc:2781] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:38.508059 19349 raft_consensus.cc:482] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:38.508113 19349 raft_consensus.cc:3037] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:38.508996 13445 raft_consensus.cc:1261] T 27d5ced54d2e4aa9b6ea01fe8700443d P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer 61b887f0d40d497badeca5604c1f0546: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:56:38.509200 19349 raft_consensus.cc:504] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:38.509645 19349 leader_election.cc:288] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 election: Requested vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:38.509959 13185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e30d88e2c054de6b87e4beb5eb14019" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770"
I20240503 13:56:38.510015 15888 raft_consensus.cc:1261] T 27d5ced54d2e4aa9b6ea01fe8700443d P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Refusing update from remote peer 61b887f0d40d497badeca5604c1f0546: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:56:38.510078 13185 raft_consensus.cc:3037] T 3e30d88e2c054de6b87e4beb5eb14019 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:38.509617 18996 consensus_queue.cc:1035] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:56:38.511078 13185 raft_consensus.cc:2454] T 3e30d88e2c054de6b87e4beb5eb14019 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 1.
I20240503 13:56:38.511337 15429 leader_election.cc:302] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61b887f0d40d497badeca5604c1f0546, 8ccf5a4ac450414c9d3579bd5800b770; no voters:
I20240503 13:56:38.511509 15888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e30d88e2c054de6b87e4beb5eb14019" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7"
I20240503 13:56:38.511619 15888 raft_consensus.cc:3037] T 3e30d88e2c054de6b87e4beb5eb14019 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:38.512668 15888 raft_consensus.cc:2454] T 3e30d88e2c054de6b87e4beb5eb14019 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 1.
I20240503 13:56:38.511691 18996 consensus_queue.cc:1035] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 13:56:38.513048 18996 raft_consensus.cc:2932] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } }
I20240503 13:56:38.513475 13445 raft_consensus.cc:2932] T 27d5ced54d2e4aa9b6ea01fe8700443d P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } }
I20240503 13:56:38.515133 15888 raft_consensus.cc:2932] T 27d5ced54d2e4aa9b6ea01fe8700443d P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } }
I20240503 13:56:38.516381 12908 catalog_manager.cc:5484] T 27d5ced54d2e4aa9b6ea01fe8700443d P 7690ee79577c45e3a2c8f8b552c0bff8 reported cstate change: config changed from index 2 to 3, VOTER ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) evicted. New cstate: current_term: 1 leader_uuid: "61b887f0d40d497badeca5604c1f0546" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } }
I20240503 13:56:38.511647 19445 raft_consensus.cc:2781] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:38.519523 19445 raft_consensus.cc:686] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 [term 1 LEADER]: Becoming Leader. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Running, Role: LEADER
I20240503 13:56:38.519857 13295 tablet_service.cc:1514] Processing DeleteTablet for tablet 27d5ced54d2e4aa9b6ea01fe8700443d with delete_type TABLET_DATA_TOMBSTONED (TS ba0194098aa84449ae17e8a0d4c43703 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:33250
I20240503 13:56:38.519791 19445 consensus_queue.cc:237] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:38.520892 19455 tablet_replica.cc:330] T 27d5ced54d2e4aa9b6ea01fe8700443d P ba0194098aa84449ae17e8a0d4c43703: stopping tablet replica
I20240503 13:56:38.521198 19455 raft_consensus.cc:2229] T 27d5ced54d2e4aa9b6ea01fe8700443d P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:38.521544 19455 raft_consensus.cc:2258] T 27d5ced54d2e4aa9b6ea01fe8700443d P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:38.521975 19455 ts_tablet_manager.cc:1875] T 27d5ced54d2e4aa9b6ea01fe8700443d P ba0194098aa84449ae17e8a0d4c43703: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:56:38.521994 12911 catalog_manager.cc:5484] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61b887f0d40d497badeca5604c1f0546 (127.6.98.69). New cstate: current_term: 1 leader_uuid: "61b887f0d40d497badeca5604c1f0546" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } }
I20240503 13:56:38.526141 19455 ts_tablet_manager.cc:1888] T 27d5ced54d2e4aa9b6ea01fe8700443d P ba0194098aa84449ae17e8a0d4c43703: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 13:56:38.526223 19455 log.cc:1196] T 27d5ced54d2e4aa9b6ea01fe8700443d P ba0194098aa84449ae17e8a0d4c43703: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/wal/wals/27d5ced54d2e4aa9b6ea01fe8700443d
I20240503 13:56:38.526772 12899 catalog_manager.cc:4830] TS ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279): tablet 27d5ced54d2e4aa9b6ea01fe8700443d (table rebalancer_extra_table_8 [id=e28614beb29c450da9e4f209b2fbe1b2]) successfully deleted
I20240503 13:56:38.582142 12904 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:56:38.593106 15868 tablet_service.cc:1467] Processing CreateTablet for tablet 0cfe856c804b45c38206c15679f15661 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=efb56c841deb4f768cf18d640487caa5]), partition=
I20240503 13:56:38.593405 15868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cfe856c804b45c38206c15679f15661. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:38.595948 13034 tablet_service.cc:1467] Processing CreateTablet for tablet 0cfe856c804b45c38206c15679f15661 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=efb56c841deb4f768cf18d640487caa5]), partition=
I20240503 13:56:38.596210 13034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cfe856c804b45c38206c15679f15661. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:38.598620 19493 tablet_bootstrap.cc:492] T 0cfe856c804b45c38206c15679f15661 P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:38.599609 19493 tablet_bootstrap.cc:654] T 0cfe856c804b45c38206c15679f15661 P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:38.600097 15605 tablet_service.cc:1467] Processing CreateTablet for tablet 0cfe856c804b45c38206c15679f15661 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=efb56c841deb4f768cf18d640487caa5]), partition=
I20240503 13:56:38.600348 15605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cfe856c804b45c38206c15679f15661. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:38.603127 19399 tablet_bootstrap.cc:492] T 0cfe856c804b45c38206c15679f15661 P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:38.604146 19399 tablet_bootstrap.cc:654] T 0cfe856c804b45c38206c15679f15661 P e47105a0f46e4511b7112e571688a4c7: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:38.604776 19400 tablet_bootstrap.cc:492] T 0cfe856c804b45c38206c15679f15661 P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:38.605705 19400 tablet_bootstrap.cc:654] T 0cfe856c804b45c38206c15679f15661 P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:38.609606 19400 tablet_bootstrap.cc:492] T 0cfe856c804b45c38206c15679f15661 P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:38.609715 19400 ts_tablet_manager.cc:1367] T 0cfe856c804b45c38206c15679f15661 P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:56:38.610199 19400 raft_consensus.cc:348] T 0cfe856c804b45c38206c15679f15661 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:38.610311 19400 raft_consensus.cc:374] T 0cfe856c804b45c38206c15679f15661 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:38.610337 19400 raft_consensus.cc:729] T 0cfe856c804b45c38206c15679f15661 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:38.610455 19400 consensus_queue.cc:260] T 0cfe856c804b45c38206c15679f15661 P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:38.610730 19400 ts_tablet_manager.cc:1398] T 0cfe856c804b45c38206c15679f15661 P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:38.615559 19399 tablet_bootstrap.cc:492] T 0cfe856c804b45c38206c15679f15661 P e47105a0f46e4511b7112e571688a4c7: No bootstrap required, opened a new log
I20240503 13:56:38.615672 19399 ts_tablet_manager.cc:1367] T 0cfe856c804b45c38206c15679f15661 P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 13:56:38.616032 19493 tablet_bootstrap.cc:492] T 0cfe856c804b45c38206c15679f15661 P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:38.616137 19493 ts_tablet_manager.cc:1367] T 0cfe856c804b45c38206c15679f15661 P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20240503 13:56:38.616161 19399 raft_consensus.cc:348] T 0cfe856c804b45c38206c15679f15661 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:38.616279 19399 raft_consensus.cc:374] T 0cfe856c804b45c38206c15679f15661 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:38.616309 19399 raft_consensus.cc:729] T 0cfe856c804b45c38206c15679f15661 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:38.616417 19399 consensus_queue.cc:260] T 0cfe856c804b45c38206c15679f15661 P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:38.616631 19493 raft_consensus.cc:348] T 0cfe856c804b45c38206c15679f15661 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:38.616711 19399 ts_tablet_manager.cc:1398] T 0cfe856c804b45c38206c15679f15661 P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:38.616740 19493 raft_consensus.cc:374] T 0cfe856c804b45c38206c15679f15661 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:38.616775 19493 raft_consensus.cc:729] T 0cfe856c804b45c38206c15679f15661 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:38.616919 19493 consensus_queue.cc:260] T 0cfe856c804b45c38206c15679f15661 P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:38.618654 19493 ts_tablet_manager.cc:1398] T 0cfe856c804b45c38206c15679f15661 P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:38.648981 15472 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:38.649468 15605 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:38.650777 13034 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:38.652581 13424 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 13:56:38.655737 15868 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 13:56:38.684405 13295 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 13:56:38.690690 15736 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:56:38.704413 13164 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:38.737460 17953 raft_consensus.cc:482] T 0cfe856c804b45c38206c15679f15661 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:38.737605 17953 raft_consensus.cc:504] T 0cfe856c804b45c38206c15679f15661 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:38.738021 17953 leader_election.cc:288] T 0cfe856c804b45c38206c15679f15661 P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951), 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:38.738360 15625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cfe856c804b45c38206c15679f15661" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d" is_pre_election: true
I20240503 13:56:38.738502 15625 raft_consensus.cc:2454] T 0cfe856c804b45c38206c15679f15661 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 0.
I20240503 13:56:38.738492 13051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cfe856c804b45c38206c15679f15661" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e" is_pre_election: true
I20240503 13:56:38.738631 13051 raft_consensus.cc:2454] T 0cfe856c804b45c38206c15679f15661 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 0.
I20240503 13:56:38.738788 15822 leader_election.cc:302] T 0cfe856c804b45c38206c15679f15661 P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:38.738971 17953 raft_consensus.cc:2781] T 0cfe856c804b45c38206c15679f15661 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:38.739019 17953 raft_consensus.cc:482] T 0cfe856c804b45c38206c15679f15661 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:38.739056 17953 raft_consensus.cc:3037] T 0cfe856c804b45c38206c15679f15661 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:38.739974 17953 raft_consensus.cc:504] T 0cfe856c804b45c38206c15679f15661 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:38.740291 17953 leader_election.cc:288] T 0cfe856c804b45c38206c15679f15661 P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 election: Requested vote from peers 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951), 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:38.740882 13051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cfe856c804b45c38206c15679f15661" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e"
I20240503 13:56:38.740998 13051 raft_consensus.cc:3037] T 0cfe856c804b45c38206c15679f15661 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:38.741003 15625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cfe856c804b45c38206c15679f15661" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d"
I20240503 13:56:38.741111 15625 raft_consensus.cc:3037] T 0cfe856c804b45c38206c15679f15661 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:38.741885 15625 raft_consensus.cc:2454] T 0cfe856c804b45c38206c15679f15661 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 1.
I20240503 13:56:38.742009 13051 raft_consensus.cc:2454] T 0cfe856c804b45c38206c15679f15661 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 1.
I20240503 13:56:38.742213 15822 leader_election.cc:302] T 0cfe856c804b45c38206c15679f15661 P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:38.744339 17953 raft_consensus.cc:2781] T 0cfe856c804b45c38206c15679f15661 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:38.744423 17953 raft_consensus.cc:686] T 0cfe856c804b45c38206c15679f15661 P e47105a0f46e4511b7112e571688a4c7 [term 1 LEADER]: Becoming Leader. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Running, Role: LEADER
I20240503 13:56:38.745096 17953 consensus_queue.cc:237] T 0cfe856c804b45c38206c15679f15661 P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:38.747606 12905 catalog_manager.cc:5484] T 0cfe856c804b45c38206c15679f15661 P e47105a0f46e4511b7112e571688a4c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e47105a0f46e4511b7112e571688a4c7 (127.6.98.72). New cstate: current_term: 1 leader_uuid: "e47105a0f46e4511b7112e571688a4c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } }
I20240503 13:56:38.764680 13314 consensus_queue.cc:237] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:38.766108 15888 raft_consensus.cc:1261] T d50d05d432e84d5eb0e54d1a68cdf158 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:56:38.766261 13051 raft_consensus.cc:1261] T d50d05d432e84d5eb0e54d1a68cdf158 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:56:38.766585 18914 consensus_queue.cc:1035] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:38.766861 18873 consensus_queue.cc:1035] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:38.769145 18914 raft_consensus.cc:2932] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:38.769587 13051 raft_consensus.cc:2932] T d50d05d432e84d5eb0e54d1a68cdf158 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:38.771911 15888 raft_consensus.cc:2932] T d50d05d432e84d5eb0e54d1a68cdf158 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:38.772430 12905 catalog_manager.cc:5484] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 reported cstate change: config changed from index 4 to 5, VOTER 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66) evicted. New cstate: current_term: 1 leader_uuid: "ba0194098aa84449ae17e8a0d4c43703" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:38.776257 13164 tablet_service.cc:1514] Processing DeleteTablet for tablet d50d05d432e84d5eb0e54d1a68cdf158 with delete_type TABLET_DATA_TOMBSTONED (TS 8ccf5a4ac450414c9d3579bd5800b770 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:41668
I20240503 13:56:38.779664 19528 tablet_replica.cc:330] T d50d05d432e84d5eb0e54d1a68cdf158 P 8ccf5a4ac450414c9d3579bd5800b770: stopping tablet replica
I20240503 13:56:38.779950 19528 raft_consensus.cc:2229] T d50d05d432e84d5eb0e54d1a68cdf158 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:38.780156 19528 raft_consensus.cc:2258] T d50d05d432e84d5eb0e54d1a68cdf158 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:38.780786 19528 ts_tablet_manager.cc:1875] T d50d05d432e84d5eb0e54d1a68cdf158 P 8ccf5a4ac450414c9d3579bd5800b770: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:56:38.783284 19528 ts_tablet_manager.cc:1888] T d50d05d432e84d5eb0e54d1a68cdf158 P 8ccf5a4ac450414c9d3579bd5800b770: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 13:56:38.783479 19528 log.cc:1196] T d50d05d432e84d5eb0e54d1a68cdf158 P 8ccf5a4ac450414c9d3579bd5800b770: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/wal/wals/d50d05d432e84d5eb0e54d1a68cdf158
I20240503 13:56:38.784191 12899 catalog_manager.cc:4830] TS 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387): tablet d50d05d432e84d5eb0e54d1a68cdf158 (table rebalance_test_table_9 [id=2fd11e7ec03a4f6891fc18482c1728a6]) successfully deleted
I20240503 13:56:38.873247 19445 consensus_queue.cc:1035] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:38.874569 15605 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:38.875051 15736 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:56:38.880250 12904 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:56:38.884682 13424 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 13:56:38.885859 13034 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:38.886430 19445 consensus_queue.cc:1035] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:38.887280 15472 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:38.891921 13164 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:38.893574 13295 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 13:56:38.899893 13051 raft_consensus.cc:1261] T 0cfe856c804b45c38206c15679f15661 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Refusing update from remote peer e47105a0f46e4511b7112e571688a4c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:38.900386 19351 consensus_queue.cc:1035] T 0cfe856c804b45c38206c15679f15661 P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:38.902487 15868 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:56:38.903973 12908 catalog_manager.cc:5576] TS 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695) does not have the latest schema for tablet 0cfe856c804b45c38206c15679f15661 (table rebalancer_dynamic_table [id=efb56c841deb4f768cf18d640487caa5]). Expected version 1 got 0
I20240503 13:56:38.906545 15624 raft_consensus.cc:1261] T 0cfe856c804b45c38206c15679f15661 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Refusing update from remote peer e47105a0f46e4511b7112e571688a4c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:38.907344 17953 consensus_queue.cc:1035] T 0cfe856c804b45c38206c15679f15661 P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:38.914920 12903 catalog_manager.cc:5576] TS 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695) does not have the latest schema for tablet 0cfe856c804b45c38206c15679f15661 (table rebalancer_dynamic_table [id=efb56c841deb4f768cf18d640487caa5]). Expected version 1 got 0
I20240503 13:56:38.917065 12904 catalog_manager.cc:5576] TS 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951) does not have the latest schema for tablet 0cfe856c804b45c38206c15679f15661 (table rebalancer_dynamic_table [id=efb56c841deb4f768cf18d640487caa5]). Expected version 1 got 0
I20240503 13:56:38.923790 12904 catalog_manager.cc:5576] TS 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951) does not have the latest schema for tablet 0cfe856c804b45c38206c15679f15661 (table rebalancer_dynamic_table [id=efb56c841deb4f768cf18d640487caa5]). Expected version 1 got 0
I20240503 13:56:38.929539 19593 tablet.cc:1722] T 0cfe856c804b45c38206c15679f15661 P 009f849ee7b54c80a3f5264695047e4d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:38.929901 19593 tablet.cc:1620] T 0cfe856c804b45c38206c15679f15661 P 009f849ee7b54c80a3f5264695047e4d: MemRowSet was empty: no flush needed.
I20240503 13:56:38.930389 19592 tablet.cc:1722] T 0cfe856c804b45c38206c15679f15661 P e47105a0f46e4511b7112e571688a4c7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:38.930684 19592 tablet.cc:1620] T 0cfe856c804b45c38206c15679f15661 P e47105a0f46e4511b7112e571688a4c7: MemRowSet was empty: no flush needed.
I20240503 13:56:38.938186 12904 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=efb56c841deb4f768cf18d640487caa5] alter complete (version 1)
I20240503 13:56:38.942102 19448 consensus_queue.cc:1035] T cdc199537531406ab832e88682140993 P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:38.952638 19595 tablet.cc:1722] T 0cfe856c804b45c38206c15679f15661 P 60ba3ffe3b7746469834cd91c84be14e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:38.953066 19595 tablet.cc:1620] T 0cfe856c804b45c38206c15679f15661 P 60ba3ffe3b7746469834cd91c84be14e: MemRowSet was empty: no flush needed.
I20240503 13:56:38.953081 19349 consensus_queue.cc:1035] T cdc199537531406ab832e88682140993 P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:38.977036 15758 consensus_queue.cc:237] T 0878ccb6135f4485b5ba35ae67a76d0c P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:38.978433 13314 raft_consensus.cc:1261] T 0878ccb6135f4485b5ba35ae67a76d0c P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Refusing update from remote peer 76c386bef61a4bd5a907266a5118ffe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:56:38.978610 13051 raft_consensus.cc:1261] T 0878ccb6135f4485b5ba35ae67a76d0c P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Refusing update from remote peer 76c386bef61a4bd5a907266a5118ffe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:56:38.979043 19615 consensus_queue.cc:1035] T 0878ccb6135f4485b5ba35ae67a76d0c P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:56:38.979225 18645 consensus_queue.cc:1035] T 0878ccb6135f4485b5ba35ae67a76d0c P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:56:38.981105 18645 raft_consensus.cc:2932] T 0878ccb6135f4485b5ba35ae67a76d0c P 76c386bef61a4bd5a907266a5118ffe2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:38.983351 13314 raft_consensus.cc:2932] T 0878ccb6135f4485b5ba35ae67a76d0c P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:38.983857 12904 catalog_manager.cc:5484] T 0878ccb6135f4485b5ba35ae67a76d0c P 76c386bef61a4bd5a907266a5118ffe2 reported cstate change: config changed from index 2 to 3, VOTER 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68) evicted. New cstate: current_term: 1 leader_uuid: "76c386bef61a4bd5a907266a5118ffe2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:38.984992 13051 raft_consensus.cc:2932] T 0878ccb6135f4485b5ba35ae67a76d0c P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:38.986583 13424 tablet_service.cc:1514] Processing DeleteTablet for tablet 0878ccb6135f4485b5ba35ae67a76d0c with delete_type TABLET_DATA_TOMBSTONED (TS 7690ee79577c45e3a2c8f8b552c0bff8 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:33678
I20240503 13:56:38.987774 19619 tablet_replica.cc:330] T 0878ccb6135f4485b5ba35ae67a76d0c P 7690ee79577c45e3a2c8f8b552c0bff8: stopping tablet replica
I20240503 13:56:38.988214 19619 raft_consensus.cc:2229] T 0878ccb6135f4485b5ba35ae67a76d0c P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:38.988590 19619 raft_consensus.cc:2258] T 0878ccb6135f4485b5ba35ae67a76d0c P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:38.989058 19619 ts_tablet_manager.cc:1875] T 0878ccb6135f4485b5ba35ae67a76d0c P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:56:38.991421 19619 ts_tablet_manager.cc:1888] T 0878ccb6135f4485b5ba35ae67a76d0c P 7690ee79577c45e3a2c8f8b552c0bff8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 13:56:38.991504 19619 log.cc:1196] T 0878ccb6135f4485b5ba35ae67a76d0c P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal/wals/0878ccb6135f4485b5ba35ae67a76d0c
I20240503 13:56:38.992064 12896 catalog_manager.cc:4830] TS 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109): tablet 0878ccb6135f4485b5ba35ae67a76d0c (table rebalancer_extra_table_0 [id=18c0e326c0a144c382cd95a8c0052057]) successfully deleted
I20240503 13:56:39.006352 19349 consensus_queue.cc:1035] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:39.010041 19448 consensus_queue.cc:1035] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:39.039520 12912 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_extra_table_26"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:56:39.045713 15472 tablet_service.cc:1467] Processing CreateTablet for tablet 72e880ee96a24cc29978a487bee7123d (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=57b3745bab3b4256bc46220a3d571b57]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:39.046016 15472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72e880ee96a24cc29978a487bee7123d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:39.045939 15736 tablet_service.cc:1467] Processing CreateTablet for tablet 72e880ee96a24cc29978a487bee7123d (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=57b3745bab3b4256bc46220a3d571b57]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:39.046241 15736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72e880ee96a24cc29978a487bee7123d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:39.046599 13424 tablet_service.cc:1467] Processing CreateTablet for tablet 6f20d5faec5940ca8b7cffa124e51e4d (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=57b3745bab3b4256bc46220a3d571b57]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:39.046844 13424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f20d5faec5940ca8b7cffa124e51e4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:39.048300 19664 tablet_bootstrap.cc:492] T 72e880ee96a24cc29978a487bee7123d P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:39.048745 15868 tablet_service.cc:1467] Processing CreateTablet for tablet 6f20d5faec5940ca8b7cffa124e51e4d (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=57b3745bab3b4256bc46220a3d571b57]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:39.049005 15868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f20d5faec5940ca8b7cffa124e51e4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:39.049201 15605 tablet_service.cc:1467] Processing CreateTablet for tablet 72e880ee96a24cc29978a487bee7123d (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=57b3745bab3b4256bc46220a3d571b57]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:39.049456 15605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72e880ee96a24cc29978a487bee7123d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:39.051108 19399 tablet_bootstrap.cc:492] T 6f20d5faec5940ca8b7cffa124e51e4d P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:39.049857 19664 tablet_bootstrap.cc:654] T 72e880ee96a24cc29978a487bee7123d P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:39.051704 15604 tablet_service.cc:1467] Processing CreateTablet for tablet 50a41096559d48a69785e6eeb534bd9c (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=57b3745bab3b4256bc46220a3d571b57]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:39.051918 15604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50a41096559d48a69785e6eeb534bd9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:39.052002 19399 tablet_bootstrap.cc:654] T 6f20d5faec5940ca8b7cffa124e51e4d P e47105a0f46e4511b7112e571688a4c7: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:39.052433 19664 tablet_bootstrap.cc:492] T 72e880ee96a24cc29978a487bee7123d P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:39.052544 19664 ts_tablet_manager.cc:1367] T 72e880ee96a24cc29978a487bee7123d P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:56:39.052975 19664 raft_consensus.cc:348] T 72e880ee96a24cc29978a487bee7123d P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:39.053077 19664 raft_consensus.cc:374] T 72e880ee96a24cc29978a487bee7123d P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:39.053107 19664 raft_consensus.cc:729] T 72e880ee96a24cc29978a487bee7123d P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:39.053213 19664 consensus_queue.cc:260] T 72e880ee96a24cc29978a487bee7123d P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:39.053261 19400 tablet_bootstrap.cc:492] T 72e880ee96a24cc29978a487bee7123d P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:39.053473 19664 ts_tablet_manager.cc:1398] T 72e880ee96a24cc29978a487bee7123d P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:39.054136 19400 tablet_bootstrap.cc:654] T 72e880ee96a24cc29978a487bee7123d P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:39.054193 13295 tablet_service.cc:1467] Processing CreateTablet for tablet 50a41096559d48a69785e6eeb534bd9c (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=57b3745bab3b4256bc46220a3d571b57]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:39.054466 13295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50a41096559d48a69785e6eeb534bd9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:39.055377 15736 tablet_service.cc:1467] Processing CreateTablet for tablet 6f20d5faec5940ca8b7cffa124e51e4d (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=57b3745bab3b4256bc46220a3d571b57]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:39.055673 15736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f20d5faec5940ca8b7cffa124e51e4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:39.056638 12903 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:56:39.058097 19664 tablet_bootstrap.cc:492] T 6f20d5faec5940ca8b7cffa124e51e4d P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:39.059002 19664 tablet_bootstrap.cc:654] T 6f20d5faec5940ca8b7cffa124e51e4d P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:39.059968 15472 tablet_service.cc:1467] Processing CreateTablet for tablet 50a41096559d48a69785e6eeb534bd9c (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=57b3745bab3b4256bc46220a3d571b57]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:39.053916 19666 tablet_bootstrap.cc:492] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:39.060238 15472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50a41096559d48a69785e6eeb534bd9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:39.061189 19666 tablet_bootstrap.cc:654] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:39.064586 19666 tablet_bootstrap.cc:492] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:39.064733 19666 ts_tablet_manager.cc:1367] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.002s
I20240503 13:56:39.065238 19666 raft_consensus.cc:348] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:39.065348 19666 raft_consensus.cc:374] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:39.065382 19666 raft_consensus.cc:729] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:39.065496 19666 consensus_queue.cc:260] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:39.065541 19665 tablet_bootstrap.cc:492] T 72e880ee96a24cc29978a487bee7123d P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:39.065770 19666 ts_tablet_manager.cc:1398] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:39.068006 19399 tablet_bootstrap.cc:492] T 6f20d5faec5940ca8b7cffa124e51e4d P e47105a0f46e4511b7112e571688a4c7: No bootstrap required, opened a new log
I20240503 13:56:39.068105 19399 ts_tablet_manager.cc:1367] T 6f20d5faec5940ca8b7cffa124e51e4d P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.001s
I20240503 13:56:39.068395 19665 tablet_bootstrap.cc:654] T 72e880ee96a24cc29978a487bee7123d P 61b887f0d40d497badeca5604c1f0546: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:39.068585 19399 raft_consensus.cc:348] T 6f20d5faec5940ca8b7cffa124e51e4d P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:39.068691 19399 raft_consensus.cc:374] T 6f20d5faec5940ca8b7cffa124e51e4d P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:39.068722 19399 raft_consensus.cc:729] T 6f20d5faec5940ca8b7cffa124e51e4d P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:39.068835 19399 consensus_queue.cc:260] T 6f20d5faec5940ca8b7cffa124e51e4d P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:39.069124 19399 ts_tablet_manager.cc:1398] T 6f20d5faec5940ca8b7cffa124e51e4d P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:39.070739 19592 tablet.cc:1722] T 0cfe856c804b45c38206c15679f15661 P e47105a0f46e4511b7112e571688a4c7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:39.071050 19592 tablet.cc:1620] T 0cfe856c804b45c38206c15679f15661 P e47105a0f46e4511b7112e571688a4c7: MemRowSet was empty: no flush needed.
I20240503 13:56:39.072368 19593 tablet.cc:1722] T 0cfe856c804b45c38206c15679f15661 P 009f849ee7b54c80a3f5264695047e4d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:39.072690 19593 tablet.cc:1620] T 0cfe856c804b45c38206c15679f15661 P 009f849ee7b54c80a3f5264695047e4d: MemRowSet was empty: no flush needed.
I20240503 13:56:39.073606 19595 tablet.cc:1722] T 0cfe856c804b45c38206c15679f15661 P 60ba3ffe3b7746469834cd91c84be14e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:39.073894 19595 tablet.cc:1620] T 0cfe856c804b45c38206c15679f15661 P 60ba3ffe3b7746469834cd91c84be14e: MemRowSet was empty: no flush needed.
I20240503 13:56:39.074379 19664 tablet_bootstrap.cc:492] T 6f20d5faec5940ca8b7cffa124e51e4d P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:39.074472 19664 ts_tablet_manager.cc:1367] T 6f20d5faec5940ca8b7cffa124e51e4d P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.000s
I20240503 13:56:39.074941 19664 raft_consensus.cc:348] T 6f20d5faec5940ca8b7cffa124e51e4d P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:39.075064 19664 raft_consensus.cc:374] T 6f20d5faec5940ca8b7cffa124e51e4d P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:39.075098 19664 raft_consensus.cc:729] T 6f20d5faec5940ca8b7cffa124e51e4d P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:39.075218 19664 consensus_queue.cc:260] T 6f20d5faec5940ca8b7cffa124e51e4d P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:39.075650 19664 ts_tablet_manager.cc:1398] T 6f20d5faec5940ca8b7cffa124e51e4d P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:39.077620 19400 tablet_bootstrap.cc:492] T 72e880ee96a24cc29978a487bee7123d P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:39.077710 19400 ts_tablet_manager.cc:1367] T 72e880ee96a24cc29978a487bee7123d P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.000s
I20240503 13:56:39.078187 19400 raft_consensus.cc:348] T 72e880ee96a24cc29978a487bee7123d P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:39.078301 19400 raft_consensus.cc:374] T 72e880ee96a24cc29978a487bee7123d P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:39.078336 19400 raft_consensus.cc:729] T 72e880ee96a24cc29978a487bee7123d P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:39.078455 19400 consensus_queue.cc:260] T 72e880ee96a24cc29978a487bee7123d P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:39.078734 19400 ts_tablet_manager.cc:1398] T 72e880ee96a24cc29978a487bee7123d P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:39.078914 19400 tablet_bootstrap.cc:492] T 50a41096559d48a69785e6eeb534bd9c P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:39.079811 19400 tablet_bootstrap.cc:654] T 50a41096559d48a69785e6eeb534bd9c P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:39.082114 19400 tablet_bootstrap.cc:492] T 50a41096559d48a69785e6eeb534bd9c P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:39.082209 19400 ts_tablet_manager.cc:1367] T 50a41096559d48a69785e6eeb534bd9c P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:56:39.082630 19400 raft_consensus.cc:348] T 50a41096559d48a69785e6eeb534bd9c P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:39.082798 19400 raft_consensus.cc:374] T 50a41096559d48a69785e6eeb534bd9c P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:39.082875 19400 raft_consensus.cc:729] T 50a41096559d48a69785e6eeb534bd9c P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:39.083242 12907 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=efb56c841deb4f768cf18d640487caa5] alter complete (version 2)
I20240503 13:56:39.083408 19400 consensus_queue.cc:260] T 50a41096559d48a69785e6eeb534bd9c P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:39.083729 19400 ts_tablet_manager.cc:1398] T 50a41096559d48a69785e6eeb534bd9c P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:39.085350 19665 tablet_bootstrap.cc:492] T 72e880ee96a24cc29978a487bee7123d P 61b887f0d40d497badeca5604c1f0546: No bootstrap required, opened a new log
I20240503 13:56:39.085517 19665 ts_tablet_manager.cc:1367] T 72e880ee96a24cc29978a487bee7123d P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20240503 13:56:39.085861 19670 tablet_bootstrap.cc:492] T 50a41096559d48a69785e6eeb534bd9c P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:39.086043 19665 raft_consensus.cc:348] T 72e880ee96a24cc29978a487bee7123d P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:39.086144 19665 raft_consensus.cc:374] T 72e880ee96a24cc29978a487bee7123d P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:39.086220 19665 raft_consensus.cc:729] T 72e880ee96a24cc29978a487bee7123d P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:39.086347 19665 consensus_queue.cc:260] T 72e880ee96a24cc29978a487bee7123d P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:39.086623 19665 ts_tablet_manager.cc:1398] T 72e880ee96a24cc29978a487bee7123d P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:39.086853 19665 tablet_bootstrap.cc:492] T 50a41096559d48a69785e6eeb534bd9c P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:39.087584 19670 tablet_bootstrap.cc:654] T 50a41096559d48a69785e6eeb534bd9c P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:39.092207 19665 tablet_bootstrap.cc:654] T 50a41096559d48a69785e6eeb534bd9c P 61b887f0d40d497badeca5604c1f0546: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:39.093276 19670 tablet_bootstrap.cc:492] T 50a41096559d48a69785e6eeb534bd9c P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:39.093385 19670 ts_tablet_manager.cc:1367] T 50a41096559d48a69785e6eeb534bd9c P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 13:56:39.093880 19665 tablet_bootstrap.cc:492] T 50a41096559d48a69785e6eeb534bd9c P 61b887f0d40d497badeca5604c1f0546: No bootstrap required, opened a new log
I20240503 13:56:39.093963 19665 ts_tablet_manager.cc:1367] T 50a41096559d48a69785e6eeb534bd9c P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 13:56:39.093875 19670 raft_consensus.cc:348] T 50a41096559d48a69785e6eeb534bd9c P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:39.094014 19670 raft_consensus.cc:374] T 50a41096559d48a69785e6eeb534bd9c P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:39.094058 19670 raft_consensus.cc:729] T 50a41096559d48a69785e6eeb534bd9c P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:39.094198 19670 consensus_queue.cc:260] T 50a41096559d48a69785e6eeb534bd9c P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:39.094422 19665 raft_consensus.cc:348] T 50a41096559d48a69785e6eeb534bd9c P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:39.094491 19670 ts_tablet_manager.cc:1398] T 50a41096559d48a69785e6eeb534bd9c P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:39.094523 19665 raft_consensus.cc:374] T 50a41096559d48a69785e6eeb534bd9c P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:39.094544 19665 raft_consensus.cc:729] T 50a41096559d48a69785e6eeb534bd9c P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:39.094643 19665 consensus_queue.cc:260] T 50a41096559d48a69785e6eeb534bd9c P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:39.094894 19665 ts_tablet_manager.cc:1398] T 50a41096559d48a69785e6eeb534bd9c P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:39.103097 19586 raft_consensus.cc:482] T 50a41096559d48a69785e6eeb534bd9c P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:39.103221 19586 raft_consensus.cc:504] T 50a41096559d48a69785e6eeb534bd9c P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:39.104434 13314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a41096559d48a69785e6eeb534bd9c" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703" is_pre_election: true
I20240503 13:56:39.104645 13314 raft_consensus.cc:2454] T 50a41096559d48a69785e6eeb534bd9c P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 0.
I20240503 13:56:39.105051 15561 leader_election.cc:302] T 50a41096559d48a69785e6eeb534bd9c P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:39.105219 18186 raft_consensus.cc:2781] T 50a41096559d48a69785e6eeb534bd9c P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:39.105257 18186 raft_consensus.cc:482] T 50a41096559d48a69785e6eeb534bd9c P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:39.105295 18186 raft_consensus.cc:3037] T 50a41096559d48a69785e6eeb534bd9c P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:39.105679 15492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a41096559d48a69785e6eeb534bd9c" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546" is_pre_election: true
I20240503 13:56:39.105810 15492 raft_consensus.cc:2454] T 50a41096559d48a69785e6eeb534bd9c P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 0.
I20240503 13:56:39.106340 18186 raft_consensus.cc:504] T 50a41096559d48a69785e6eeb534bd9c P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:39.106678 18186 leader_election.cc:288] T 50a41096559d48a69785e6eeb534bd9c P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 election: Requested vote from peers ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279), 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153)
I20240503 13:56:39.105262 19586 leader_election.cc:288] T 50a41096559d48a69785e6eeb534bd9c P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279), 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153)
I20240503 13:56:39.107827 15492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a41096559d48a69785e6eeb534bd9c" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546"
I20240503 13:56:39.107949 15492 raft_consensus.cc:3037] T 50a41096559d48a69785e6eeb534bd9c P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:39.108971 15492 raft_consensus.cc:2454] T 50a41096559d48a69785e6eeb534bd9c P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 1.
I20240503 13:56:39.109258 15561 leader_election.cc:302] T 50a41096559d48a69785e6eeb534bd9c P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 61b887f0d40d497badeca5604c1f0546; no voters:
I20240503 13:56:39.109736 19586 raft_consensus.cc:2781] T 50a41096559d48a69785e6eeb534bd9c P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:39.109848 19586 raft_consensus.cc:686] T 50a41096559d48a69785e6eeb534bd9c P 009f849ee7b54c80a3f5264695047e4d [term 1 LEADER]: Becoming Leader. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Running, Role: LEADER
I20240503 13:56:39.109985 19586 consensus_queue.cc:237] T 50a41096559d48a69785e6eeb534bd9c P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:39.112138 13314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a41096559d48a69785e6eeb534bd9c" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703"
I20240503 13:56:39.112269 13314 raft_consensus.cc:3037] T 50a41096559d48a69785e6eeb534bd9c P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:39.112586 12906 catalog_manager.cc:5484] T 50a41096559d48a69785e6eeb534bd9c P 009f849ee7b54c80a3f5264695047e4d reported cstate change: term changed from 0 to 1, leader changed from <none> to 009f849ee7b54c80a3f5264695047e4d (127.6.98.70). New cstate: current_term: 1 leader_uuid: "009f849ee7b54c80a3f5264695047e4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:39.113399 13314 raft_consensus.cc:2454] T 50a41096559d48a69785e6eeb534bd9c P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 1.
I20240503 13:56:39.113843 13424 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 13:56:39.119060 13034 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:39.120059 13164 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:39.123886 15604 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:39.129844 15472 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:39.130090 15868 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:56:39.140753 15736 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:56:39.155485 19373 raft_consensus.cc:482] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:39.155598 19373 raft_consensus.cc:504] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:39.155953 19373 leader_election.cc:288] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:39.156399 15888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f20d5faec5940ca8b7cffa124e51e4d" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7" is_pre_election: true
I20240503 13:56:39.156538 15888 raft_consensus.cc:2454] T 6f20d5faec5940ca8b7cffa124e51e4d P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 0.
I20240503 13:56:39.156991 13380 leader_election.cc:302] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:39.157312 19373 raft_consensus.cc:2781] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:39.157290 15758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f20d5faec5940ca8b7cffa124e51e4d" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2" is_pre_election: true
I20240503 13:56:39.157358 19373 raft_consensus.cc:482] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:39.157390 19373 raft_consensus.cc:3037] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:39.157416 15758 raft_consensus.cc:2454] T 6f20d5faec5940ca8b7cffa124e51e4d P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 0.
I20240503 13:56:39.158471 19373 raft_consensus.cc:504] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:39.158773 19373 leader_election.cc:288] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 election: Requested vote from peers 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:39.159097 15888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f20d5faec5940ca8b7cffa124e51e4d" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7"
I20240503 13:56:39.159263 15888 raft_consensus.cc:3037] T 6f20d5faec5940ca8b7cffa124e51e4d P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:39.159351 15758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f20d5faec5940ca8b7cffa124e51e4d" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2"
I20240503 13:56:39.159451 15758 raft_consensus.cc:3037] T 6f20d5faec5940ca8b7cffa124e51e4d P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:39.160218 15888 raft_consensus.cc:2454] T 6f20d5faec5940ca8b7cffa124e51e4d P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 1.
I20240503 13:56:39.160281 15758 raft_consensus.cc:2454] T 6f20d5faec5940ca8b7cffa124e51e4d P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 1.
I20240503 13:56:39.160557 13383 leader_election.cc:302] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, 76c386bef61a4bd5a907266a5118ffe2; no voters:
I20240503 13:56:39.160715 19373 raft_consensus.cc:2781] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:39.160797 19373 raft_consensus.cc:686] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 LEADER]: Becoming Leader. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Running, Role: LEADER
I20240503 13:56:39.160923 19373 consensus_queue.cc:237] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:39.168735 12906 catalog_manager.cc:5484] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68). New cstate: current_term: 1 leader_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:39.172544 13295 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 13:56:39.185480 12906 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:56:39.193445 19592 tablet.cc:1722] T 0cfe856c804b45c38206c15679f15661 P e47105a0f46e4511b7112e571688a4c7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:39.198343 19593 tablet.cc:1722] T 0cfe856c804b45c38206c15679f15661 P 009f849ee7b54c80a3f5264695047e4d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:39.200347 12903 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table_new_name [id=efb56c841deb4f768cf18d640487caa5] alter complete (version 3)
I20240503 13:56:39.200400 19595 tablet.cc:1722] T 0cfe856c804b45c38206c15679f15661 P 60ba3ffe3b7746469834cd91c84be14e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:39.228117 18186 raft_consensus.cc:482] T 72e880ee96a24cc29978a487bee7123d P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:39.228255 18186 raft_consensus.cc:504] T 72e880ee96a24cc29978a487bee7123d P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:39.228696 18186 leader_election.cc:288] T 72e880ee96a24cc29978a487bee7123d P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153), 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885)
I20240503 13:56:39.229203 15758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e880ee96a24cc29978a487bee7123d" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2" is_pre_election: true
I20240503 13:56:39.229251 15492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e880ee96a24cc29978a487bee7123d" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546" is_pre_election: true
I20240503 13:56:39.229342 15758 raft_consensus.cc:2454] T 72e880ee96a24cc29978a487bee7123d P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 0.
I20240503 13:56:39.229372 15492 raft_consensus.cc:2454] T 72e880ee96a24cc29978a487bee7123d P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 0.
I20240503 13:56:39.229725 15561 leader_election.cc:302] T 72e880ee96a24cc29978a487bee7123d P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 61b887f0d40d497badeca5604c1f0546; no voters:
I20240503 13:56:39.230029 18186 raft_consensus.cc:2781] T 72e880ee96a24cc29978a487bee7123d P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:39.230082 18186 raft_consensus.cc:482] T 72e880ee96a24cc29978a487bee7123d P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:39.230116 18186 raft_consensus.cc:3037] T 72e880ee96a24cc29978a487bee7123d P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:39.232142 18186 raft_consensus.cc:504] T 72e880ee96a24cc29978a487bee7123d P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:39.232440 18186 leader_election.cc:288] T 72e880ee96a24cc29978a487bee7123d P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 election: Requested vote from peers 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153), 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885)
I20240503 13:56:39.232570 13314 consensus_queue.cc:237] T dc544319be0b484f8e9e5a3fbc40f530 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:39.233021 15758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e880ee96a24cc29978a487bee7123d" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2"
I20240503 13:56:39.233135 15758 raft_consensus.cc:3037] T 72e880ee96a24cc29978a487bee7123d P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:39.233139 15492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e880ee96a24cc29978a487bee7123d" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546"
I20240503 13:56:39.233248 15492 raft_consensus.cc:3037] T 72e880ee96a24cc29978a487bee7123d P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:39.234107 15758 raft_consensus.cc:2454] T 72e880ee96a24cc29978a487bee7123d P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 1.
I20240503 13:56:39.234184 15492 raft_consensus.cc:2454] T 72e880ee96a24cc29978a487bee7123d P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 1.
I20240503 13:56:39.234486 15561 leader_election.cc:302] T 72e880ee96a24cc29978a487bee7123d P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 76c386bef61a4bd5a907266a5118ffe2; no voters:
I20240503 13:56:39.234640 18186 raft_consensus.cc:2781] T 72e880ee96a24cc29978a487bee7123d P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:39.234723 18186 raft_consensus.cc:686] T 72e880ee96a24cc29978a487bee7123d P 009f849ee7b54c80a3f5264695047e4d [term 1 LEADER]: Becoming Leader. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Running, Role: LEADER
I20240503 13:56:39.234862 18186 consensus_queue.cc:237] T 72e880ee96a24cc29978a487bee7123d P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:39.235677 13051 raft_consensus.cc:1261] T dc544319be0b484f8e9e5a3fbc40f530 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:56:39.236155 18914 consensus_queue.cc:1035] T dc544319be0b484f8e9e5a3fbc40f530 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:39.236999 12906 catalog_manager.cc:5484] T 72e880ee96a24cc29978a487bee7123d P 009f849ee7b54c80a3f5264695047e4d reported cstate change: term changed from 0 to 1, leader changed from <none> to 009f849ee7b54c80a3f5264695047e4d (127.6.98.70). New cstate: current_term: 1 leader_uuid: "009f849ee7b54c80a3f5264695047e4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:39.239706 15758 raft_consensus.cc:1261] T dc544319be0b484f8e9e5a3fbc40f530 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:56:39.240118 18914 consensus_queue.cc:1035] T dc544319be0b484f8e9e5a3fbc40f530 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:39.240476 19527 raft_consensus.cc:2932] T dc544319be0b484f8e9e5a3fbc40f530 P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:39.243232 15758 raft_consensus.cc:2932] T dc544319be0b484f8e9e5a3fbc40f530 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:39.243553 12906 catalog_manager.cc:5484] T dc544319be0b484f8e9e5a3fbc40f530 P ba0194098aa84449ae17e8a0d4c43703 reported cstate change: config changed from index 4 to 5, VOTER 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66) evicted. New cstate: current_term: 1 leader_uuid: "ba0194098aa84449ae17e8a0d4c43703" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:39.243585 13051 raft_consensus.cc:2932] T dc544319be0b484f8e9e5a3fbc40f530 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:39.247671 13164 tablet_service.cc:1514] Processing DeleteTablet for tablet dc544319be0b484f8e9e5a3fbc40f530 with delete_type TABLET_DATA_TOMBSTONED (TS 8ccf5a4ac450414c9d3579bd5800b770 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:41668
I20240503 13:56:39.247979 19528 tablet_replica.cc:330] T dc544319be0b484f8e9e5a3fbc40f530 P 8ccf5a4ac450414c9d3579bd5800b770: stopping tablet replica
I20240503 13:56:39.248144 19528 raft_consensus.cc:2229] T dc544319be0b484f8e9e5a3fbc40f530 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:39.248312 19528 raft_consensus.cc:2258] T dc544319be0b484f8e9e5a3fbc40f530 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:39.248970 19528 ts_tablet_manager.cc:1875] T dc544319be0b484f8e9e5a3fbc40f530 P 8ccf5a4ac450414c9d3579bd5800b770: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:56:39.250710 19528 ts_tablet_manager.cc:1888] T dc544319be0b484f8e9e5a3fbc40f530 P 8ccf5a4ac450414c9d3579bd5800b770: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 13:56:39.250777 19528 log.cc:1196] T dc544319be0b484f8e9e5a3fbc40f530 P 8ccf5a4ac450414c9d3579bd5800b770: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/wal/wals/dc544319be0b484f8e9e5a3fbc40f530
I20240503 13:56:39.251253 12899 catalog_manager.cc:4830] TS 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387): tablet dc544319be0b484f8e9e5a3fbc40f530 (table rebalance_test_table_19 [id=e6609c16048348029c01374cb5a68ce8]) successfully deleted
I20240503 13:56:39.302469 12906 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:39.302626 12906 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:39.304883 12906 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Sending DeleteTablet for 3 replicas of tablet 0cfe856c804b45c38206c15679f15661
I20240503 13:56:39.305439 15868 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cfe856c804b45c38206c15679f15661 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:39 UTC) from {username='slave'} at 127.0.0.1:50498
I20240503 13:56:39.305572 13034 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cfe856c804b45c38206c15679f15661 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:39 UTC) from {username='slave'} at 127.0.0.1:59458
I20240503 13:56:39.308593 15604 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cfe856c804b45c38206c15679f15661 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:39 UTC) from {username='slave'} at 127.0.0.1:60718
I20240503 13:56:39.309005 19754 tablet_replica.cc:330] T 0cfe856c804b45c38206c15679f15661 P 009f849ee7b54c80a3f5264695047e4d: stopping tablet replica
I20240503 13:56:39.309221 19754 raft_consensus.cc:2229] T 0cfe856c804b45c38206c15679f15661 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:39.309377 19754 raft_consensus.cc:2258] T 0cfe856c804b45c38206c15679f15661 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:39.309924 19754 ts_tablet_manager.cc:1875] T 0cfe856c804b45c38206c15679f15661 P 009f849ee7b54c80a3f5264695047e4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:39.312312 19754 ts_tablet_manager.cc:1888] T 0cfe856c804b45c38206c15679f15661 P 009f849ee7b54c80a3f5264695047e4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:39.312451 19754 log.cc:1196] T 0cfe856c804b45c38206c15679f15661 P 009f849ee7b54c80a3f5264695047e4d: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/wal/wals/0cfe856c804b45c38206c15679f15661
I20240503 13:56:39.312721 19754 ts_tablet_manager.cc:1909] T 0cfe856c804b45c38206c15679f15661 P 009f849ee7b54c80a3f5264695047e4d: Deleting consensus metadata
I20240503 13:56:39.313452 12896 catalog_manager.cc:4830] TS 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951): tablet 0cfe856c804b45c38206c15679f15661 (table rebalancer_dynamic_table_new_name [id=efb56c841deb4f768cf18d640487caa5]) successfully deleted
I20240503 13:56:39.329070 19752 tablet_replica.cc:330] T 0cfe856c804b45c38206c15679f15661 P e47105a0f46e4511b7112e571688a4c7: stopping tablet replica
I20240503 13:56:39.329274 19752 raft_consensus.cc:2229] T 0cfe856c804b45c38206c15679f15661 P e47105a0f46e4511b7112e571688a4c7 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:56:39.329521 19752 raft_consensus.cc:2258] T 0cfe856c804b45c38206c15679f15661 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:39.330365 19752 ts_tablet_manager.cc:1875] T 0cfe856c804b45c38206c15679f15661 P e47105a0f46e4511b7112e571688a4c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:39.333029 19752 ts_tablet_manager.cc:1888] T 0cfe856c804b45c38206c15679f15661 P e47105a0f46e4511b7112e571688a4c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:39.334406 19753 tablet_replica.cc:330] T 0cfe856c804b45c38206c15679f15661 P 60ba3ffe3b7746469834cd91c84be14e: stopping tablet replica
I20240503 13:56:39.334678 19753 raft_consensus.cc:2229] T 0cfe856c804b45c38206c15679f15661 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:39.334898 19753 raft_consensus.cc:2258] T 0cfe856c804b45c38206c15679f15661 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:39.335367 19752 log.cc:1196] T 0cfe856c804b45c38206c15679f15661 P e47105a0f46e4511b7112e571688a4c7: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-7/wal/wals/0cfe856c804b45c38206c15679f15661
I20240503 13:56:39.336086 19753 ts_tablet_manager.cc:1875] T 0cfe856c804b45c38206c15679f15661 P 60ba3ffe3b7746469834cd91c84be14e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:39.336416 19752 ts_tablet_manager.cc:1909] T 0cfe856c804b45c38206c15679f15661 P e47105a0f46e4511b7112e571688a4c7: Deleting consensus metadata
I20240503 13:56:39.337127 12896 catalog_manager.cc:4830] TS e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): tablet 0cfe856c804b45c38206c15679f15661 (table rebalancer_dynamic_table_new_name [id=efb56c841deb4f768cf18d640487caa5]) successfully deleted
I20240503 13:56:39.345638 13034 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:39.348325 15736 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:56:39.350266 19753 ts_tablet_manager.cc:1888] T 0cfe856c804b45c38206c15679f15661 P 60ba3ffe3b7746469834cd91c84be14e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:39.350991 19753 log.cc:1196] T 0cfe856c804b45c38206c15679f15661 P 60ba3ffe3b7746469834cd91c84be14e: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal/wals/0cfe856c804b45c38206c15679f15661
I20240503 13:56:39.351429 19753 ts_tablet_manager.cc:1909] T 0cfe856c804b45c38206c15679f15661 P 60ba3ffe3b7746469834cd91c84be14e: Deleting consensus metadata
I20240503 13:56:39.352079 15604 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 13:56:39.354511 13424 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:56:39.355391 12897 catalog_manager.cc:4830] TS 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695): tablet 0cfe856c804b45c38206c15679f15661 (table rebalancer_dynamic_table_new_name [id=efb56c841deb4f768cf18d640487caa5]) successfully deleted
I20240503 13:56:39.364401 15472 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:39.375119 15868 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 13:56:39.381857 13295 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 13:56:39.385385 13164 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:39.410283 12911 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:56:39.416119 15472 tablet_service.cc:1467] Processing CreateTablet for tablet 3c0439254bd345c8b161947ab56c3bab (DEFAULT_TABLE table=rebalancer_dynamic_table [id=dc317e5192db4ebfb68b121ce2ff3bb9]), partition=
I20240503 13:56:39.416409 15472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c0439254bd345c8b161947ab56c3bab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:39.417174 15604 tablet_service.cc:1467] Processing CreateTablet for tablet 3c0439254bd345c8b161947ab56c3bab (DEFAULT_TABLE table=rebalancer_dynamic_table [id=dc317e5192db4ebfb68b121ce2ff3bb9]), partition=
I20240503 13:56:39.417421 15604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c0439254bd345c8b161947ab56c3bab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:39.417670 13295 tablet_service.cc:1467] Processing CreateTablet for tablet 3c0439254bd345c8b161947ab56c3bab (DEFAULT_TABLE table=rebalancer_dynamic_table [id=dc317e5192db4ebfb68b121ce2ff3bb9]), partition=
I20240503 13:56:39.417902 13295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c0439254bd345c8b161947ab56c3bab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:39.418658 19665 tablet_bootstrap.cc:492] T 3c0439254bd345c8b161947ab56c3bab P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:39.419607 19665 tablet_bootstrap.cc:654] T 3c0439254bd345c8b161947ab56c3bab P 61b887f0d40d497badeca5604c1f0546: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:39.420032 19670 tablet_bootstrap.cc:492] T 3c0439254bd345c8b161947ab56c3bab P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:39.420239 19400 tablet_bootstrap.cc:492] T 3c0439254bd345c8b161947ab56c3bab P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:39.420873 19670 tablet_bootstrap.cc:654] T 3c0439254bd345c8b161947ab56c3bab P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:39.421191 19400 tablet_bootstrap.cc:654] T 3c0439254bd345c8b161947ab56c3bab P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:39.423233 19400 tablet_bootstrap.cc:492] T 3c0439254bd345c8b161947ab56c3bab P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:39.423308 19400 ts_tablet_manager.cc:1367] T 3c0439254bd345c8b161947ab56c3bab P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:56:39.423722 19400 raft_consensus.cc:348] T 3c0439254bd345c8b161947ab56c3bab P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:39.423832 19400 raft_consensus.cc:374] T 3c0439254bd345c8b161947ab56c3bab P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:39.423872 19400 raft_consensus.cc:729] T 3c0439254bd345c8b161947ab56c3bab P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:39.423985 19400 consensus_queue.cc:260] T 3c0439254bd345c8b161947ab56c3bab P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:39.424237 19400 ts_tablet_manager.cc:1398] T 3c0439254bd345c8b161947ab56c3bab P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:39.424337 19670 tablet_bootstrap.cc:492] T 3c0439254bd345c8b161947ab56c3bab P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:39.424424 19670 ts_tablet_manager.cc:1367] T 3c0439254bd345c8b161947ab56c3bab P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 13:56:39.424916 19670 raft_consensus.cc:348] T 3c0439254bd345c8b161947ab56c3bab P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:39.425021 19670 raft_consensus.cc:374] T 3c0439254bd345c8b161947ab56c3bab P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:39.425051 19670 raft_consensus.cc:729] T 3c0439254bd345c8b161947ab56c3bab P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:39.425177 19670 consensus_queue.cc:260] T 3c0439254bd345c8b161947ab56c3bab P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:39.425436 19670 ts_tablet_manager.cc:1398] T 3c0439254bd345c8b161947ab56c3bab P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:39.427506 19665 tablet_bootstrap.cc:492] T 3c0439254bd345c8b161947ab56c3bab P 61b887f0d40d497badeca5604c1f0546: No bootstrap required, opened a new log
I20240503 13:56:39.427596 19665 ts_tablet_manager.cc:1367] T 3c0439254bd345c8b161947ab56c3bab P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 13:56:39.428040 19665 raft_consensus.cc:348] T 3c0439254bd345c8b161947ab56c3bab P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:39.428143 19665 raft_consensus.cc:374] T 3c0439254bd345c8b161947ab56c3bab P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:39.428174 19665 raft_consensus.cc:729] T 3c0439254bd345c8b161947ab56c3bab P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:39.428279 19665 consensus_queue.cc:260] T 3c0439254bd345c8b161947ab56c3bab P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:39.428537 19665 ts_tablet_manager.cc:1398] T 3c0439254bd345c8b161947ab56c3bab P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:39.458117 13051 consensus_queue.cc:237] T 855c5fc4cefb48b58533f26d62595465 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:39.459316 15623 raft_consensus.cc:1261] T 855c5fc4cefb48b58533f26d62595465 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Refusing update from remote peer 60ba3ffe3b7746469834cd91c84be14e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:56:39.459352 13445 raft_consensus.cc:1261] T 855c5fc4cefb48b58533f26d62595465 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer 60ba3ffe3b7746469834cd91c84be14e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:56:39.459723 19691 consensus_queue.cc:1035] T 855c5fc4cefb48b58533f26d62595465 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:56:39.460080 19307 consensus_queue.cc:1035] T 855c5fc4cefb48b58533f26d62595465 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:56:39.461898 19307 raft_consensus.cc:2932] T 855c5fc4cefb48b58533f26d62595465 P 60ba3ffe3b7746469834cd91c84be14e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:39.462368 13445 raft_consensus.cc:2932] T 855c5fc4cefb48b58533f26d62595465 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:39.462414 15624 raft_consensus.cc:2932] T 855c5fc4cefb48b58533f26d62595465 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:39.465811 12911 catalog_manager.cc:5484] T 855c5fc4cefb48b58533f26d62595465 P 60ba3ffe3b7746469834cd91c84be14e reported cstate change: config changed from index 2 to 3, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) evicted. New cstate: current_term: 1 leader_uuid: "60ba3ffe3b7746469834cd91c84be14e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:39.468238 15868 tablet_service.cc:1514] Processing DeleteTablet for tablet 855c5fc4cefb48b58533f26d62595465 with delete_type TABLET_DATA_TOMBSTONED (TS e47105a0f46e4511b7112e571688a4c7 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:50498
I20240503 13:56:39.468600 19752 tablet_replica.cc:330] T 855c5fc4cefb48b58533f26d62595465 P e47105a0f46e4511b7112e571688a4c7: stopping tablet replica
I20240503 13:56:39.468734 19752 raft_consensus.cc:2229] T 855c5fc4cefb48b58533f26d62595465 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:39.468842 19752 raft_consensus.cc:2258] T 855c5fc4cefb48b58533f26d62595465 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:39.469303 19752 ts_tablet_manager.cc:1875] T 855c5fc4cefb48b58533f26d62595465 P e47105a0f46e4511b7112e571688a4c7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:56:39.471169 19752 ts_tablet_manager.cc:1888] T 855c5fc4cefb48b58533f26d62595465 P e47105a0f46e4511b7112e571688a4c7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 13:56:39.471225 19752 log.cc:1196] T 855c5fc4cefb48b58533f26d62595465 P e47105a0f46e4511b7112e571688a4c7: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-7/wal/wals/855c5fc4cefb48b58533f26d62595465
I20240503 13:56:39.471740 12896 catalog_manager.cc:4830] TS e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): tablet 855c5fc4cefb48b58533f26d62595465 (table rebalancer_extra_table_10 [id=658f6c781f344047b9273985ceb17229]) successfully deleted
I20240503 13:56:39.541266 18873 raft_consensus.cc:482] T 3c0439254bd345c8b161947ab56c3bab P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:39.541404 18873 raft_consensus.cc:504] T 3c0439254bd345c8b161947ab56c3bab P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:39.541846 18873 leader_election.cc:288] T 3c0439254bd345c8b161947ab56c3bab P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153), 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951)
I20240503 13:56:39.542418 15624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c0439254bd345c8b161947ab56c3bab" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d" is_pre_election: true
I20240503 13:56:39.542408 15492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c0439254bd345c8b161947ab56c3bab" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546" is_pre_election: true
I20240503 13:56:39.542543 15624 raft_consensus.cc:2454] T 3c0439254bd345c8b161947ab56c3bab P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 0.
I20240503 13:56:39.542580 15492 raft_consensus.cc:2454] T 3c0439254bd345c8b161947ab56c3bab P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 0.
I20240503 13:56:39.542835 13249 leader_election.cc:302] T 3c0439254bd345c8b161947ab56c3bab P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:39.542994 18873 raft_consensus.cc:2781] T 3c0439254bd345c8b161947ab56c3bab P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:39.543043 18873 raft_consensus.cc:482] T 3c0439254bd345c8b161947ab56c3bab P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:39.543085 18873 raft_consensus.cc:3037] T 3c0439254bd345c8b161947ab56c3bab P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:39.544039 18873 raft_consensus.cc:504] T 3c0439254bd345c8b161947ab56c3bab P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:39.544371 18873 leader_election.cc:288] T 3c0439254bd345c8b161947ab56c3bab P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: Requested vote from peers 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153), 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951)
I20240503 13:56:39.544745 15492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c0439254bd345c8b161947ab56c3bab" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546"
I20240503 13:56:39.544867 15492 raft_consensus.cc:3037] T 3c0439254bd345c8b161947ab56c3bab P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:39.545605 15492 raft_consensus.cc:2454] T 3c0439254bd345c8b161947ab56c3bab P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 1.
I20240503 13:56:39.545895 15624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c0439254bd345c8b161947ab56c3bab" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d"
I20240503 13:56:39.546016 15624 raft_consensus.cc:3037] T 3c0439254bd345c8b161947ab56c3bab P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:39.546720 15624 raft_consensus.cc:2454] T 3c0439254bd345c8b161947ab56c3bab P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 1.
I20240503 13:56:39.547004 13249 leader_election.cc:302] T 3c0439254bd345c8b161947ab56c3bab P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:39.547650 18873 raft_consensus.cc:2781] T 3c0439254bd345c8b161947ab56c3bab P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:39.547744 18873 raft_consensus.cc:686] T 3c0439254bd345c8b161947ab56c3bab P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Becoming Leader. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Running, Role: LEADER
I20240503 13:56:39.547883 18873 consensus_queue.cc:237] T 3c0439254bd345c8b161947ab56c3bab P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:39.549794 12909 catalog_manager.cc:5484] T 3c0439254bd345c8b161947ab56c3bab P ba0194098aa84449ae17e8a0d4c43703 reported cstate change: term changed from 0 to 1, leader changed from <none> to ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67). New cstate: current_term: 1 leader_uuid: "ba0194098aa84449ae17e8a0d4c43703" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:39.567716 15736 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:56:39.567829 15604 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 13:56:39.569909 13034 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:39.585784 13424 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:56:39.587257 15472 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:39.588142 13295 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 13:56:39.590917 13164 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:39.593273 15868 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 13:56:39.628271 19586 consensus_queue.cc:1035] T 50a41096559d48a69785e6eeb534bd9c P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:39.632865 18186 consensus_queue.cc:1035] T 50a41096559d48a69785e6eeb534bd9c P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:39.637198 19373 consensus_queue.cc:1035] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:39.641698 19402 consensus_queue.cc:1035] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:39.653177 13051 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "66121c347e6c443dac7c5123826a7ae1"
dest_uuid: "60ba3ffe3b7746469834cd91c84be14e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51480
I20240503 13:56:39.653347 13051 raft_consensus.cc:595] T 66121c347e6c443dac7c5123826a7ae1 P 60ba3ffe3b7746469834cd91c84be14e [term 1 LEADER]: Received request to transfer leadership
I20240503 13:56:39.681609 12909 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:56:39.685601 15624 raft_consensus.cc:1261] T 3c0439254bd345c8b161947ab56c3bab P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:39.685683 15492 raft_consensus.cc:1261] T 3c0439254bd345c8b161947ab56c3bab P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:39.686170 18873 consensus_queue.cc:1035] T 3c0439254bd345c8b161947ab56c3bab P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:39.686342 18914 consensus_queue.cc:1035] T 3c0439254bd345c8b161947ab56c3bab P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:39.689327 19593 tablet.cc:1722] T 3c0439254bd345c8b161947ab56c3bab P 009f849ee7b54c80a3f5264695047e4d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:39.689374 19875 tablet.cc:1722] T 3c0439254bd345c8b161947ab56c3bab P 61b887f0d40d497badeca5604c1f0546: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:39.689622 12911 catalog_manager.cc:5576] TS 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951) does not have the latest schema for tablet 3c0439254bd345c8b161947ab56c3bab (table rebalancer_dynamic_table [id=dc317e5192db4ebfb68b121ce2ff3bb9]). Expected version 1 got 0
I20240503 13:56:39.689666 19593 tablet.cc:1620] T 3c0439254bd345c8b161947ab56c3bab P 009f849ee7b54c80a3f5264695047e4d: MemRowSet was empty: no flush needed.
I20240503 13:56:39.689890 19875 tablet.cc:1620] T 3c0439254bd345c8b161947ab56c3bab P 61b887f0d40d497badeca5604c1f0546: MemRowSet was empty: no flush needed.
I20240503 13:56:39.692510 12911 catalog_manager.cc:5576] TS 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153) does not have the latest schema for tablet 3c0439254bd345c8b161947ab56c3bab (table rebalancer_dynamic_table [id=dc317e5192db4ebfb68b121ce2ff3bb9]). Expected version 1 got 0
I20240503 13:56:39.694193 19874 tablet.cc:1722] T 3c0439254bd345c8b161947ab56c3bab P ba0194098aa84449ae17e8a0d4c43703: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:39.694582 19874 tablet.cc:1620] T 3c0439254bd345c8b161947ab56c3bab P ba0194098aa84449ae17e8a0d4c43703: MemRowSet was empty: no flush needed.
I20240503 13:56:39.695797 12904 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=dc317e5192db4ebfb68b121ce2ff3bb9] alter complete (version 1)
I20240503 13:56:39.696648 18186 consensus_queue.cc:1035] T 72e880ee96a24cc29978a487bee7123d P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:39.701579 19586 consensus_queue.cc:1035] T 72e880ee96a24cc29978a487bee7123d P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:39.702118 19307 raft_consensus.cc:979] T 66121c347e6c443dac7c5123826a7ae1 P 60ba3ffe3b7746469834cd91c84be14e: : Instructing follower 61b887f0d40d497badeca5604c1f0546 to start an election
I20240503 13:56:39.702239 19691 raft_consensus.cc:1067] T 66121c347e6c443dac7c5123826a7ae1 P 60ba3ffe3b7746469834cd91c84be14e [term 1 LEADER]: Signalling peer 61b887f0d40d497badeca5604c1f0546 to start an election
I20240503 13:56:39.704146 15492 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "66121c347e6c443dac7c5123826a7ae1"
dest_uuid: "61b887f0d40d497badeca5604c1f0546"
from {username='slave'} at 127.6.98.65:37605
I20240503 13:56:39.704268 15492 raft_consensus.cc:482] T 66121c347e6c443dac7c5123826a7ae1 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 13:56:39.704349 15492 raft_consensus.cc:3037] T 66121c347e6c443dac7c5123826a7ae1 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:56:39.705427 15492 raft_consensus.cc:504] T 66121c347e6c443dac7c5123826a7ae1 P 61b887f0d40d497badeca5604c1f0546 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:39.705907 15492 leader_election.cc:288] T 66121c347e6c443dac7c5123826a7ae1 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 2 election: Requested vote from peers 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695), ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:39.706245 13314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66121c347e6c443dac7c5123826a7ae1" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "ba0194098aa84449ae17e8a0d4c43703"
I20240503 13:56:39.706369 13314 raft_consensus.cc:3037] T 66121c347e6c443dac7c5123826a7ae1 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:56:39.706405 13183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66121c347e6c443dac7c5123826a7ae1" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770"
I20240503 13:56:39.706519 13183 raft_consensus.cc:3037] T 66121c347e6c443dac7c5123826a7ae1 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:56:39.706915 13051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66121c347e6c443dac7c5123826a7ae1" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "60ba3ffe3b7746469834cd91c84be14e"
I20240503 13:56:39.707036 13051 raft_consensus.cc:3032] T 66121c347e6c443dac7c5123826a7ae1 P 60ba3ffe3b7746469834cd91c84be14e [term 1 LEADER]: Stepping down as leader of term 1
I20240503 13:56:39.707075 13051 raft_consensus.cc:729] T 66121c347e6c443dac7c5123826a7ae1 P 60ba3ffe3b7746469834cd91c84be14e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Running, Role: LEADER
I20240503 13:56:39.707224 13051 consensus_queue.cc:260] T 66121c347e6c443dac7c5123826a7ae1 P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:39.707329 13314 raft_consensus.cc:2454] T 66121c347e6c443dac7c5123826a7ae1 P ba0194098aa84449ae17e8a0d4c43703 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 2.
I20240503 13:56:39.707378 13051 raft_consensus.cc:3037] T 66121c347e6c443dac7c5123826a7ae1 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:56:39.707384 13183 raft_consensus.cc:2454] T 66121c347e6c443dac7c5123826a7ae1 P 8ccf5a4ac450414c9d3579bd5800b770 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 2.
I20240503 13:56:39.707724 15429 leader_election.cc:302] T 66121c347e6c443dac7c5123826a7ae1 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 61b887f0d40d497badeca5604c1f0546, 8ccf5a4ac450414c9d3579bd5800b770, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:39.708328 13051 raft_consensus.cc:2454] T 66121c347e6c443dac7c5123826a7ae1 P 60ba3ffe3b7746469834cd91c84be14e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 2.
I20240503 13:56:39.708537 19828 raft_consensus.cc:2781] T 66121c347e6c443dac7c5123826a7ae1 P 61b887f0d40d497badeca5604c1f0546 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:56:39.708635 19828 raft_consensus.cc:686] T 66121c347e6c443dac7c5123826a7ae1 P 61b887f0d40d497badeca5604c1f0546 [term 2 LEADER]: Becoming Leader. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Running, Role: LEADER
I20240503 13:56:39.708758 19828 consensus_queue.cc:237] T 66121c347e6c443dac7c5123826a7ae1 P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:39.713362 12911 catalog_manager.cc:5484] T 66121c347e6c443dac7c5123826a7ae1 P 61b887f0d40d497badeca5604c1f0546 reported cstate change: term changed from 1 to 2, leader changed from 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65) to 61b887f0d40d497badeca5604c1f0546 (127.6.98.69). New cstate: current_term: 2 leader_uuid: "61b887f0d40d497badeca5604c1f0546" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:39.747356 12907 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_extra_table_28"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:56:39.758004 15604 tablet_service.cc:1467] Processing CreateTablet for tablet fff6b70fd6094e2ea86b03ff34b2f09b (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=9f47d08d451443dbb03bd467d7ea8427]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:39.758054 15472 tablet_service.cc:1467] Processing CreateTablet for tablet fff6b70fd6094e2ea86b03ff34b2f09b (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=9f47d08d451443dbb03bd467d7ea8427]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:39.758344 15472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fff6b70fd6094e2ea86b03ff34b2f09b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:39.758365 15604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fff6b70fd6094e2ea86b03ff34b2f09b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:39.758323 15736 tablet_service.cc:1467] Processing CreateTablet for tablet f21a51b44e0c40389e68f9e88c427bae (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=9f47d08d451443dbb03bd467d7ea8427]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:39.758607 15736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f21a51b44e0c40389e68f9e88c427bae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:39.760293 19400 tablet_bootstrap.cc:492] T fff6b70fd6094e2ea86b03ff34b2f09b P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:39.760903 13034 tablet_service.cc:1467] Processing CreateTablet for tablet fff6b70fd6094e2ea86b03ff34b2f09b (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=9f47d08d451443dbb03bd467d7ea8427]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:39.761183 13034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fff6b70fd6094e2ea86b03ff34b2f09b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:39.761297 19400 tablet_bootstrap.cc:654] T fff6b70fd6094e2ea86b03ff34b2f09b P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:39.762220 15868 tablet_service.cc:1467] Processing CreateTablet for tablet f21a51b44e0c40389e68f9e88c427bae (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=9f47d08d451443dbb03bd467d7ea8427]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:39.762519 15868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f21a51b44e0c40389e68f9e88c427bae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:39.763187 19911 tablet_bootstrap.cc:492] T fff6b70fd6094e2ea86b03ff34b2f09b P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:39.763347 13032 tablet_service.cc:1467] Processing CreateTablet for tablet 2f2c47eed62b4bec89714eb6f293ce7b (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=9f47d08d451443dbb03bd467d7ea8427]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:39.763568 13032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f2c47eed62b4bec89714eb6f293ce7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:39.764070 19911 tablet_bootstrap.cc:654] T fff6b70fd6094e2ea86b03ff34b2f09b P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:39.764261 15604 tablet_service.cc:1467] Processing CreateTablet for tablet f21a51b44e0c40389e68f9e88c427bae (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=9f47d08d451443dbb03bd467d7ea8427]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:39.764515 15604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f21a51b44e0c40389e68f9e88c427bae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:39.762220 15867 tablet_service.cc:1467] Processing CreateTablet for tablet 2f2c47eed62b4bec89714eb6f293ce7b (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=9f47d08d451443dbb03bd467d7ea8427]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:39.765103 15867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f2c47eed62b4bec89714eb6f293ce7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:39.765731 19665 tablet_bootstrap.cc:492] T fff6b70fd6094e2ea86b03ff34b2f09b P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:39.766722 19665 tablet_bootstrap.cc:654] T fff6b70fd6094e2ea86b03ff34b2f09b P 61b887f0d40d497badeca5604c1f0546: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:39.770085 19913 tablet_bootstrap.cc:492] T f21a51b44e0c40389e68f9e88c427bae P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:39.770324 15472 tablet_service.cc:1467] Processing CreateTablet for tablet 2f2c47eed62b4bec89714eb6f293ce7b (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=9f47d08d451443dbb03bd467d7ea8427]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:39.770589 15472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f2c47eed62b4bec89714eb6f293ce7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:39.771342 19913 tablet_bootstrap.cc:654] T f21a51b44e0c40389e68f9e88c427bae P e47105a0f46e4511b7112e571688a4c7: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:39.771428 15604 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 13:56:39.774551 19665 tablet_bootstrap.cc:492] T fff6b70fd6094e2ea86b03ff34b2f09b P 61b887f0d40d497badeca5604c1f0546: No bootstrap required, opened a new log
I20240503 13:56:39.774650 19665 ts_tablet_manager.cc:1367] T fff6b70fd6094e2ea86b03ff34b2f09b P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 13:56:39.775135 19400 tablet_bootstrap.cc:492] T fff6b70fd6094e2ea86b03ff34b2f09b P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:39.775228 19400 ts_tablet_manager.cc:1367] T fff6b70fd6094e2ea86b03ff34b2f09b P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240503 13:56:39.775144 19665 raft_consensus.cc:348] T fff6b70fd6094e2ea86b03ff34b2f09b P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:39.775276 19665 raft_consensus.cc:374] T fff6b70fd6094e2ea86b03ff34b2f09b P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:39.775310 19665 raft_consensus.cc:729] T fff6b70fd6094e2ea86b03ff34b2f09b P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:39.775437 19665 consensus_queue.cc:260] T fff6b70fd6094e2ea86b03ff34b2f09b P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:39.775673 19400 raft_consensus.cc:348] T fff6b70fd6094e2ea86b03ff34b2f09b P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:39.775740 19665 ts_tablet_manager.cc:1398] T fff6b70fd6094e2ea86b03ff34b2f09b P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:39.775768 19400 raft_consensus.cc:374] T fff6b70fd6094e2ea86b03ff34b2f09b P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:39.775813 19400 raft_consensus.cc:729] T fff6b70fd6094e2ea86b03ff34b2f09b P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:39.775899 19665 tablet_bootstrap.cc:492] T 2f2c47eed62b4bec89714eb6f293ce7b P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:39.775914 19400 consensus_queue.cc:260] T fff6b70fd6094e2ea86b03ff34b2f09b P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:39.776166 19400 ts_tablet_manager.cc:1398] T fff6b70fd6094e2ea86b03ff34b2f09b P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:39.776326 19400 tablet_bootstrap.cc:492] T f21a51b44e0c40389e68f9e88c427bae P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:39.777266 19400 tablet_bootstrap.cc:654] T f21a51b44e0c40389e68f9e88c427bae P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:39.779692 19917 tablet_bootstrap.cc:492] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:39.781069 19917 tablet_bootstrap.cc:654] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:39.782229 15472 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:39.782545 19917 tablet_bootstrap.cc:492] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:39.782635 19917 ts_tablet_manager.cc:1367] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:56:39.782883 19349 raft_consensus.cc:482] T fff6b70fd6094e2ea86b03ff34b2f09b P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:39.782958 19349 raft_consensus.cc:504] T fff6b70fd6094e2ea86b03ff34b2f09b P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:39.783097 19917 raft_consensus.cc:348] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:39.783205 19917 raft_consensus.cc:374] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:39.783223 19665 tablet_bootstrap.cc:654] T 2f2c47eed62b4bec89714eb6f293ce7b P 61b887f0d40d497badeca5604c1f0546: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:39.783237 19917 raft_consensus.cc:729] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:39.783336 19349 leader_election.cc:288] T fff6b70fd6094e2ea86b03ff34b2f09b P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695), 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951)
I20240503 13:56:39.783353 19917 consensus_queue.cc:260] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:39.783639 19917 ts_tablet_manager.cc:1398] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:39.784585 13051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fff6b70fd6094e2ea86b03ff34b2f09b" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e" is_pre_election: true
I20240503 13:56:39.785331 19665 tablet_bootstrap.cc:492] T 2f2c47eed62b4bec89714eb6f293ce7b P 61b887f0d40d497badeca5604c1f0546: No bootstrap required, opened a new log
I20240503 13:56:39.785411 19665 ts_tablet_manager.cc:1367] T 2f2c47eed62b4bec89714eb6f293ce7b P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 13:56:39.785879 19665 raft_consensus.cc:348] T 2f2c47eed62b4bec89714eb6f293ce7b P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:39.785979 19665 raft_consensus.cc:374] T 2f2c47eed62b4bec89714eb6f293ce7b P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:39.786010 19665 raft_consensus.cc:729] T 2f2c47eed62b4bec89714eb6f293ce7b P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:39.786168 19665 consensus_queue.cc:260] T 2f2c47eed62b4bec89714eb6f293ce7b P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
W20240503 13:56:39.786281 15427 leader_election.cc:341] T fff6b70fd6094e2ea86b03ff34b2f09b P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695): Illegal state: must be running to vote when last-logged opid is not known
I20240503 13:56:39.787176 19665 ts_tablet_manager.cc:1398] T 2f2c47eed62b4bec89714eb6f293ce7b P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:39.788692 15624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fff6b70fd6094e2ea86b03ff34b2f09b" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d" is_pre_election: true
I20240503 13:56:39.788836 15624 raft_consensus.cc:2454] T fff6b70fd6094e2ea86b03ff34b2f09b P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 0.
I20240503 13:56:39.791765 13032 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:56:39.796721 15426 leader_election.cc:302] T fff6b70fd6094e2ea86b03ff34b2f09b P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 61b887f0d40d497badeca5604c1f0546; no voters: 60ba3ffe3b7746469834cd91c84be14e
I20240503 13:56:39.797078 19349 raft_consensus.cc:2781] T fff6b70fd6094e2ea86b03ff34b2f09b P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:39.797137 19349 raft_consensus.cc:482] T fff6b70fd6094e2ea86b03ff34b2f09b P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:39.797173 19349 raft_consensus.cc:3037] T fff6b70fd6094e2ea86b03ff34b2f09b P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:39.798195 19349 raft_consensus.cc:504] T fff6b70fd6094e2ea86b03ff34b2f09b P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:39.798556 19349 leader_election.cc:288] T fff6b70fd6094e2ea86b03ff34b2f09b P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 election: Requested vote from peers 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695), 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951)
I20240503 13:56:39.799422 15867 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 13:56:39.799583 15624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fff6b70fd6094e2ea86b03ff34b2f09b" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d"
I20240503 13:56:39.799712 15624 raft_consensus.cc:3037] T fff6b70fd6094e2ea86b03ff34b2f09b P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:39.800652 15624 raft_consensus.cc:2454] T fff6b70fd6094e2ea86b03ff34b2f09b P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 1.
I20240503 13:56:39.800637 13051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fff6b70fd6094e2ea86b03ff34b2f09b" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e"
W20240503 13:56:39.801040 15427 leader_election.cc:341] T fff6b70fd6094e2ea86b03ff34b2f09b P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695): Illegal state: must be running to vote when last-logged opid is not known
I20240503 13:56:39.801131 12907 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:56:39.801259 15426 leader_election.cc:302] T fff6b70fd6094e2ea86b03ff34b2f09b P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 61b887f0d40d497badeca5604c1f0546; no voters: 60ba3ffe3b7746469834cd91c84be14e
I20240503 13:56:39.801618 19828 raft_consensus.cc:2781] T fff6b70fd6094e2ea86b03ff34b2f09b P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:39.801677 19828 raft_consensus.cc:686] T fff6b70fd6094e2ea86b03ff34b2f09b P 61b887f0d40d497badeca5604c1f0546 [term 1 LEADER]: Becoming Leader. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Running, Role: LEADER
I20240503 13:56:39.801784 19828 consensus_queue.cc:237] T fff6b70fd6094e2ea86b03ff34b2f09b P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:39.806748 12904 catalog_manager.cc:5484] T fff6b70fd6094e2ea86b03ff34b2f09b P 61b887f0d40d497badeca5604c1f0546 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61b887f0d40d497badeca5604c1f0546 (127.6.98.69). New cstate: current_term: 1 leader_uuid: "61b887f0d40d497badeca5604c1f0546" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:39.807993 19400 tablet_bootstrap.cc:492] T f21a51b44e0c40389e68f9e88c427bae P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:39.808099 19400 ts_tablet_manager.cc:1367] T f21a51b44e0c40389e68f9e88c427bae P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.000s
I20240503 13:56:39.808801 13164 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:39.809059 19874 tablet.cc:1722] T 3c0439254bd345c8b161947ab56c3bab P ba0194098aa84449ae17e8a0d4c43703: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:39.809379 19874 tablet.cc:1620] T 3c0439254bd345c8b161947ab56c3bab P ba0194098aa84449ae17e8a0d4c43703: MemRowSet was empty: no flush needed.
I20240503 13:56:39.809195 19400 raft_consensus.cc:348] T f21a51b44e0c40389e68f9e88c427bae P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:39.809572 19400 raft_consensus.cc:374] T f21a51b44e0c40389e68f9e88c427bae P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:39.809608 19400 raft_consensus.cc:729] T f21a51b44e0c40389e68f9e88c427bae P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:39.809725 19400 consensus_queue.cc:260] T f21a51b44e0c40389e68f9e88c427bae P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:39.810163 19400 ts_tablet_manager.cc:1398] T f21a51b44e0c40389e68f9e88c427bae P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:39.810364 19875 tablet.cc:1722] T 3c0439254bd345c8b161947ab56c3bab P 61b887f0d40d497badeca5604c1f0546: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:39.810626 19875 tablet.cc:1620] T 3c0439254bd345c8b161947ab56c3bab P 61b887f0d40d497badeca5604c1f0546: MemRowSet was empty: no flush needed.
I20240503 13:56:39.812896 19593 tablet.cc:1722] T 3c0439254bd345c8b161947ab56c3bab P 009f849ee7b54c80a3f5264695047e4d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:39.813169 19593 tablet.cc:1620] T 3c0439254bd345c8b161947ab56c3bab P 009f849ee7b54c80a3f5264695047e4d: MemRowSet was empty: no flush needed.
I20240503 13:56:39.818639 12909 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=dc317e5192db4ebfb68b121ce2ff3bb9] alter complete (version 2)
I20240503 13:56:39.822877 19911 tablet_bootstrap.cc:492] T fff6b70fd6094e2ea86b03ff34b2f09b P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:39.822984 19911 ts_tablet_manager.cc:1367] T fff6b70fd6094e2ea86b03ff34b2f09b P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.060s user 0.000s sys 0.002s
I20240503 13:56:39.823436 19911 raft_consensus.cc:348] T fff6b70fd6094e2ea86b03ff34b2f09b P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:39.823559 19911 raft_consensus.cc:374] T fff6b70fd6094e2ea86b03ff34b2f09b P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:39.823587 19911 raft_consensus.cc:729] T fff6b70fd6094e2ea86b03ff34b2f09b P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:39.823714 19911 consensus_queue.cc:260] T fff6b70fd6094e2ea86b03ff34b2f09b P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:39.823971 19911 ts_tablet_manager.cc:1398] T fff6b70fd6094e2ea86b03ff34b2f09b P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:39.824122 19911 tablet_bootstrap.cc:492] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:39.825114 19911 tablet_bootstrap.cc:654] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:39.828390 19913 tablet_bootstrap.cc:492] T f21a51b44e0c40389e68f9e88c427bae P e47105a0f46e4511b7112e571688a4c7: No bootstrap required, opened a new log
I20240503 13:56:39.828593 19913 ts_tablet_manager.cc:1367] T f21a51b44e0c40389e68f9e88c427bae P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.059s user 0.001s sys 0.001s
I20240503 13:56:39.829012 13424 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:56:39.829213 19913 raft_consensus.cc:348] T f21a51b44e0c40389e68f9e88c427bae P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:39.829398 19913 raft_consensus.cc:374] T f21a51b44e0c40389e68f9e88c427bae P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:39.829473 19913 raft_consensus.cc:729] T f21a51b44e0c40389e68f9e88c427bae P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:39.829571 19911 tablet_bootstrap.cc:492] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:39.829668 19911 ts_tablet_manager.cc:1367] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240503 13:56:39.829756 19913 consensus_queue.cc:260] T f21a51b44e0c40389e68f9e88c427bae P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:39.830096 19911 raft_consensus.cc:348] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:39.830209 19911 raft_consensus.cc:374] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:39.830205 19913 ts_tablet_manager.cc:1398] T f21a51b44e0c40389e68f9e88c427bae P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:56:39.830235 19911 raft_consensus.cc:729] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:39.830345 19911 consensus_queue.cc:260] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:39.830516 19913 tablet_bootstrap.cc:492] T 2f2c47eed62b4bec89714eb6f293ce7b P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:39.830574 19911 ts_tablet_manager.cc:1398] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:39.831830 15736 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:56:39.835052 19913 tablet_bootstrap.cc:654] T 2f2c47eed62b4bec89714eb6f293ce7b P e47105a0f46e4511b7112e571688a4c7: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:39.841045 13295 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 13:56:39.849124 19913 tablet_bootstrap.cc:492] T 2f2c47eed62b4bec89714eb6f293ce7b P e47105a0f46e4511b7112e571688a4c7: No bootstrap required, opened a new log
I20240503 13:56:39.849246 19913 ts_tablet_manager.cc:1367] T 2f2c47eed62b4bec89714eb6f293ce7b P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.002s
I20240503 13:56:39.849779 19913 raft_consensus.cc:348] T 2f2c47eed62b4bec89714eb6f293ce7b P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:39.849889 19913 raft_consensus.cc:374] T 2f2c47eed62b4bec89714eb6f293ce7b P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:39.849917 19913 raft_consensus.cc:729] T 2f2c47eed62b4bec89714eb6f293ce7b P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:39.850082 19913 consensus_queue.cc:260] T 2f2c47eed62b4bec89714eb6f293ce7b P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:39.850431 19913 ts_tablet_manager.cc:1398] T 2f2c47eed62b4bec89714eb6f293ce7b P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:39.856750 19307 raft_consensus.cc:482] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:39.856897 19307 raft_consensus.cc:504] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:39.857266 19307 leader_election.cc:288] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:39.857564 15492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f2c47eed62b4bec89714eb6f293ce7b" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546" is_pre_election: true
I20240503 13:56:39.857690 15492 raft_consensus.cc:2454] T 2f2c47eed62b4bec89714eb6f293ce7b P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 0.
I20240503 13:56:39.857889 15888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f2c47eed62b4bec89714eb6f293ce7b" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7" is_pre_election: true
I20240503 13:56:39.858008 15888 raft_consensus.cc:2454] T 2f2c47eed62b4bec89714eb6f293ce7b P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 0.
I20240503 13:56:39.858275 12987 leader_election.cc:302] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:39.858814 19307 raft_consensus.cc:2781] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:39.858866 19307 raft_consensus.cc:482] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:39.858898 19307 raft_consensus.cc:3037] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:39.859930 19307 raft_consensus.cc:504] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:39.860271 19307 leader_election.cc:288] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 election: Requested vote from peers 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:39.860986 15888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f2c47eed62b4bec89714eb6f293ce7b" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7"
I20240503 13:56:39.861099 15888 raft_consensus.cc:3037] T 2f2c47eed62b4bec89714eb6f293ce7b P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:39.862126 15888 raft_consensus.cc:2454] T 2f2c47eed62b4bec89714eb6f293ce7b P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 1.
I20240503 13:56:39.862332 15492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f2c47eed62b4bec89714eb6f293ce7b" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546"
I20240503 13:56:39.862444 15492 raft_consensus.cc:3037] T 2f2c47eed62b4bec89714eb6f293ce7b P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:39.862962 12987 leader_election.cc:302] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:39.863174 19307 raft_consensus.cc:2781] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:39.863250 19307 raft_consensus.cc:686] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e [term 1 LEADER]: Becoming Leader. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Running, Role: LEADER
I20240503 13:56:39.863373 19307 consensus_queue.cc:237] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:39.864464 15492 raft_consensus.cc:2454] T 2f2c47eed62b4bec89714eb6f293ce7b P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 1.
I20240503 13:56:39.865700 12904 catalog_manager.cc:5484] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e reported cstate change: term changed from 0 to 1, leader changed from <none> to 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65). New cstate: current_term: 1 leader_uuid: "60ba3ffe3b7746469834cd91c84be14e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } }
I20240503 13:56:39.922379 19841 raft_consensus.cc:482] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:39.922519 19841 raft_consensus.cc:504] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:39.922960 19841 leader_election.cc:288] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:39.922957 12907 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:56:39.923542 15624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21a51b44e0c40389e68f9e88c427bae" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d" is_pre_election: true
I20240503 13:56:39.923557 15888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21a51b44e0c40389e68f9e88c427bae" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7" is_pre_election: true
I20240503 13:56:39.923677 15624 raft_consensus.cc:2454] T f21a51b44e0c40389e68f9e88c427bae P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 0.
I20240503 13:56:39.923691 15888 raft_consensus.cc:2454] T f21a51b44e0c40389e68f9e88c427bae P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 0.
I20240503 13:56:39.924161 15691 leader_election.cc:302] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 76c386bef61a4bd5a907266a5118ffe2; no voters:
I20240503 13:56:39.924690 19841 raft_consensus.cc:2781] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:39.924754 19841 raft_consensus.cc:482] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:39.924794 19841 raft_consensus.cc:3037] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:39.925979 19841 raft_consensus.cc:504] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:39.926339 19841 leader_election.cc:288] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 election: Requested vote from peers 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:39.927125 15624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21a51b44e0c40389e68f9e88c427bae" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d"
I20240503 13:56:39.927254 15624 raft_consensus.cc:3037] T f21a51b44e0c40389e68f9e88c427bae P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:39.928736 19874 tablet.cc:1722] T 3c0439254bd345c8b161947ab56c3bab P ba0194098aa84449ae17e8a0d4c43703: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:39.929128 19875 tablet.cc:1722] T 3c0439254bd345c8b161947ab56c3bab P 61b887f0d40d497badeca5604c1f0546: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:39.930672 15888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21a51b44e0c40389e68f9e88c427bae" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7"
I20240503 13:56:39.930799 15888 raft_consensus.cc:3037] T f21a51b44e0c40389e68f9e88c427bae P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:39.931731 15888 raft_consensus.cc:2454] T f21a51b44e0c40389e68f9e88c427bae P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 1.
I20240503 13:56:39.932193 15691 leader_election.cc:302] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76c386bef61a4bd5a907266a5118ffe2, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:39.932971 15624 raft_consensus.cc:2454] T f21a51b44e0c40389e68f9e88c427bae P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 1.
I20240503 13:56:39.933038 19593 tablet.cc:1722] T 3c0439254bd345c8b161947ab56c3bab P 009f849ee7b54c80a3f5264695047e4d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:39.934271 12907 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table_new_name [id=dc317e5192db4ebfb68b121ce2ff3bb9] alter complete (version 3)
I20240503 13:56:39.936998 19841 raft_consensus.cc:2781] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:39.937095 19841 raft_consensus.cc:686] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 [term 1 LEADER]: Becoming Leader. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Running, Role: LEADER
I20240503 13:56:39.937228 19841 consensus_queue.cc:237] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:39.939491 12907 catalog_manager.cc:5484] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71). New cstate: current_term: 1 leader_uuid: "76c386bef61a4bd5a907266a5118ffe2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:39.998354 13032 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:40.001271 15604 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 13:56:40.002645 15472 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 13:56:40.005437 13424 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:56:40.005448 15867 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 13:56:40.007143 15736 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:56:40.034914 13295 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 13:56:40.036204 13164 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:40.038738 12912 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:40.038902 12912 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:40.041121 12912 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Sending DeleteTablet for 3 replicas of tablet 3c0439254bd345c8b161947ab56c3bab
I20240503 13:56:40.042007 15604 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c0439254bd345c8b161947ab56c3bab with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:40 UTC) from {username='slave'} at 127.0.0.1:60718
I20240503 13:56:40.042366 19977 tablet_replica.cc:330] T 3c0439254bd345c8b161947ab56c3bab P 009f849ee7b54c80a3f5264695047e4d: stopping tablet replica
I20240503 13:56:40.042582 19977 raft_consensus.cc:2229] T 3c0439254bd345c8b161947ab56c3bab P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:40.042794 13295 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c0439254bd345c8b161947ab56c3bab with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:40 UTC) from {username='slave'} at 127.0.0.1:33250
I20240503 13:56:40.042833 19977 raft_consensus.cc:2258] T 3c0439254bd345c8b161947ab56c3bab P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:40.043264 15472 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c0439254bd345c8b161947ab56c3bab with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:40 UTC) from {username='slave'} at 127.0.0.1:58758
I20240503 13:56:40.043663 19977 ts_tablet_manager.cc:1875] T 3c0439254bd345c8b161947ab56c3bab P 009f849ee7b54c80a3f5264695047e4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:40.046118 19977 ts_tablet_manager.cc:1888] T 3c0439254bd345c8b161947ab56c3bab P 009f849ee7b54c80a3f5264695047e4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:40.046254 19977 log.cc:1196] T 3c0439254bd345c8b161947ab56c3bab P 009f849ee7b54c80a3f5264695047e4d: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/wal/wals/3c0439254bd345c8b161947ab56c3bab
I20240503 13:56:40.046602 19977 ts_tablet_manager.cc:1909] T 3c0439254bd345c8b161947ab56c3bab P 009f849ee7b54c80a3f5264695047e4d: Deleting consensus metadata
I20240503 13:56:40.047408 12896 catalog_manager.cc:4830] TS 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951): tablet 3c0439254bd345c8b161947ab56c3bab (table rebalancer_dynamic_table_new_name [id=dc317e5192db4ebfb68b121ce2ff3bb9]) successfully deleted
I20240503 13:56:40.047523 19978 tablet_replica.cc:330] T 3c0439254bd345c8b161947ab56c3bab P ba0194098aa84449ae17e8a0d4c43703: stopping tablet replica
I20240503 13:56:40.047668 19978 raft_consensus.cc:2229] T 3c0439254bd345c8b161947ab56c3bab P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:56:40.047859 19978 raft_consensus.cc:2258] T 3c0439254bd345c8b161947ab56c3bab P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:40.048385 19978 ts_tablet_manager.cc:1875] T 3c0439254bd345c8b161947ab56c3bab P ba0194098aa84449ae17e8a0d4c43703: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:40.050592 19978 ts_tablet_manager.cc:1888] T 3c0439254bd345c8b161947ab56c3bab P ba0194098aa84449ae17e8a0d4c43703: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:40.050668 19978 log.cc:1196] T 3c0439254bd345c8b161947ab56c3bab P ba0194098aa84449ae17e8a0d4c43703: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/wal/wals/3c0439254bd345c8b161947ab56c3bab
I20240503 13:56:40.050948 19978 ts_tablet_manager.cc:1909] T 3c0439254bd345c8b161947ab56c3bab P ba0194098aa84449ae17e8a0d4c43703: Deleting consensus metadata
I20240503 13:56:40.051607 12899 catalog_manager.cc:4830] TS ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279): tablet 3c0439254bd345c8b161947ab56c3bab (table rebalancer_dynamic_table_new_name [id=dc317e5192db4ebfb68b121ce2ff3bb9]) successfully deleted
I20240503 13:56:40.052018 19979 tablet_replica.cc:330] T 3c0439254bd345c8b161947ab56c3bab P 61b887f0d40d497badeca5604c1f0546: stopping tablet replica
I20240503 13:56:40.052179 19979 raft_consensus.cc:2229] T 3c0439254bd345c8b161947ab56c3bab P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:40.052307 19979 raft_consensus.cc:2258] T 3c0439254bd345c8b161947ab56c3bab P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:40.055233 19979 ts_tablet_manager.cc:1875] T 3c0439254bd345c8b161947ab56c3bab P 61b887f0d40d497badeca5604c1f0546: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:40.057582 19979 ts_tablet_manager.cc:1888] T 3c0439254bd345c8b161947ab56c3bab P 61b887f0d40d497badeca5604c1f0546: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:40.057736 19979 log.cc:1196] T 3c0439254bd345c8b161947ab56c3bab P 61b887f0d40d497badeca5604c1f0546: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/wal/wals/3c0439254bd345c8b161947ab56c3bab
I20240503 13:56:40.058054 19979 ts_tablet_manager.cc:1909] T 3c0439254bd345c8b161947ab56c3bab P 61b887f0d40d497badeca5604c1f0546: Deleting consensus metadata
I20240503 13:56:40.058853 12899 catalog_manager.cc:4830] TS 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153): tablet 3c0439254bd345c8b161947ab56c3bab (table rebalancer_dynamic_table_new_name [id=dc317e5192db4ebfb68b121ce2ff3bb9]) successfully deleted
I20240503 13:56:40.101210 13183 consensus_queue.cc:237] T a3c4cbf779554d8b84a48de78a9c392e P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:40.102520 13051 raft_consensus.cc:1261] T a3c4cbf779554d8b84a48de78a9c392e P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Refusing update from remote peer 8ccf5a4ac450414c9d3579bd5800b770: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:56:40.102520 15492 raft_consensus.cc:1261] T a3c4cbf779554d8b84a48de78a9c392e P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Refusing update from remote peer 8ccf5a4ac450414c9d3579bd5800b770: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:56:40.102937 19424 consensus_queue.cc:1035] T a3c4cbf779554d8b84a48de78a9c392e P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:56:40.103149 19019 consensus_queue.cc:1035] T a3c4cbf779554d8b84a48de78a9c392e P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:56:40.105235 19998 raft_consensus.cc:2932] T a3c4cbf779554d8b84a48de78a9c392e P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } }
I20240503 13:56:40.105773 13051 raft_consensus.cc:2932] T a3c4cbf779554d8b84a48de78a9c392e P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } }
I20240503 13:56:40.105773 15492 raft_consensus.cc:2932] T a3c4cbf779554d8b84a48de78a9c392e P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } }
I20240503 13:56:40.109333 12912 catalog_manager.cc:5484] T a3c4cbf779554d8b84a48de78a9c392e P 60ba3ffe3b7746469834cd91c84be14e reported cstate change: config changed from index 2 to 3, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) evicted. New cstate: current_term: 1 leader_uuid: "8ccf5a4ac450414c9d3579bd5800b770" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } }
I20240503 13:56:40.113358 15867 tablet_service.cc:1514] Processing DeleteTablet for tablet a3c4cbf779554d8b84a48de78a9c392e with delete_type TABLET_DATA_TOMBSTONED (TS e47105a0f46e4511b7112e571688a4c7 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:50498
I20240503 13:56:40.113737 20003 tablet_replica.cc:330] T a3c4cbf779554d8b84a48de78a9c392e P e47105a0f46e4511b7112e571688a4c7: stopping tablet replica
I20240503 13:56:40.113967 20003 raft_consensus.cc:2229] T a3c4cbf779554d8b84a48de78a9c392e P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:40.114109 20003 raft_consensus.cc:2258] T a3c4cbf779554d8b84a48de78a9c392e P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:40.114574 20003 ts_tablet_manager.cc:1875] T a3c4cbf779554d8b84a48de78a9c392e P e47105a0f46e4511b7112e571688a4c7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:56:40.116663 19307 raft_consensus.cc:482] T fff6b70fd6094e2ea86b03ff34b2f09b P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:40.116783 19307 raft_consensus.cc:504] T fff6b70fd6094e2ea86b03ff34b2f09b P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:40.117098 20003 ts_tablet_manager.cc:1888] T a3c4cbf779554d8b84a48de78a9c392e P e47105a0f46e4511b7112e571688a4c7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 13:56:40.117228 20003 log.cc:1196] T a3c4cbf779554d8b84a48de78a9c392e P e47105a0f46e4511b7112e571688a4c7: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-7/wal/wals/a3c4cbf779554d8b84a48de78a9c392e
I20240503 13:56:40.117242 19307 leader_election.cc:288] T fff6b70fd6094e2ea86b03ff34b2f09b P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153), 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951)
I20240503 13:56:40.117677 15624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fff6b70fd6094e2ea86b03ff34b2f09b" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d" is_pre_election: true
I20240503 13:56:40.117785 15492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fff6b70fd6094e2ea86b03ff34b2f09b" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546" is_pre_election: true
I20240503 13:56:40.117852 15624 raft_consensus.cc:2379] T fff6b70fd6094e2ea86b03ff34b2f09b P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60ba3ffe3b7746469834cd91c84be14e in current term 1: Already voted for candidate 61b887f0d40d497badeca5604c1f0546 in this term.
I20240503 13:56:40.118111 12896 catalog_manager.cc:4830] TS e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): tablet a3c4cbf779554d8b84a48de78a9c392e (table rebalancer_extra_table_2 [id=65fb8aeaab8a4eb884e47e4e0f7ef87d]) successfully deleted
I20240503 13:56:40.118449 12990 leader_election.cc:302] T fff6b70fd6094e2ea86b03ff34b2f09b P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e; no voters: 009f849ee7b54c80a3f5264695047e4d, 61b887f0d40d497badeca5604c1f0546
I20240503 13:56:40.118870 19307 raft_consensus.cc:3037] T fff6b70fd6094e2ea86b03ff34b2f09b P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:40.119894 19307 raft_consensus.cc:2726] T fff6b70fd6094e2ea86b03ff34b2f09b P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 13:56:40.142757 12908 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:56:40.148239 15604 tablet_service.cc:1467] Processing CreateTablet for tablet a9522b81d1604459a46f2f5433ca4970 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3d3c742fef2046d6861c48cf78324535]), partition=
I20240503 13:56:40.148284 13424 tablet_service.cc:1467] Processing CreateTablet for tablet a9522b81d1604459a46f2f5433ca4970 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3d3c742fef2046d6861c48cf78324535]), partition=
I20240503 13:56:40.148571 15604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9522b81d1604459a46f2f5433ca4970. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:40.148576 13424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9522b81d1604459a46f2f5433ca4970. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:40.150893 20029 tablet_bootstrap.cc:492] T a9522b81d1604459a46f2f5433ca4970 P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:40.151057 19400 tablet_bootstrap.cc:492] T a9522b81d1604459a46f2f5433ca4970 P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:40.151989 19400 tablet_bootstrap.cc:654] T a9522b81d1604459a46f2f5433ca4970 P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:40.152058 20029 tablet_bootstrap.cc:654] T a9522b81d1604459a46f2f5433ca4970 P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:40.152668 15472 tablet_service.cc:1467] Processing CreateTablet for tablet a9522b81d1604459a46f2f5433ca4970 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3d3c742fef2046d6861c48cf78324535]), partition=
I20240503 13:56:40.152890 15472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9522b81d1604459a46f2f5433ca4970. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:40.153867 20029 tablet_bootstrap.cc:492] T a9522b81d1604459a46f2f5433ca4970 P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:40.154079 20029 ts_tablet_manager.cc:1367] T a9522b81d1604459a46f2f5433ca4970 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 13:56:40.154549 20029 raft_consensus.cc:348] T a9522b81d1604459a46f2f5433ca4970 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:40.154659 20029 raft_consensus.cc:374] T a9522b81d1604459a46f2f5433ca4970 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:40.154686 20029 raft_consensus.cc:729] T a9522b81d1604459a46f2f5433ca4970 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:40.154801 20029 consensus_queue.cc:260] T a9522b81d1604459a46f2f5433ca4970 P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:40.155122 20029 ts_tablet_manager.cc:1398] T a9522b81d1604459a46f2f5433ca4970 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:40.155603 19665 tablet_bootstrap.cc:492] T a9522b81d1604459a46f2f5433ca4970 P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:40.156544 19665 tablet_bootstrap.cc:654] T a9522b81d1604459a46f2f5433ca4970 P 61b887f0d40d497badeca5604c1f0546: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:40.158782 19400 tablet_bootstrap.cc:492] T a9522b81d1604459a46f2f5433ca4970 P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:40.158874 19400 ts_tablet_manager.cc:1367] T a9522b81d1604459a46f2f5433ca4970 P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240503 13:56:40.159307 19400 raft_consensus.cc:348] T a9522b81d1604459a46f2f5433ca4970 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:40.159421 19400 raft_consensus.cc:374] T a9522b81d1604459a46f2f5433ca4970 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:40.159456 19400 raft_consensus.cc:729] T a9522b81d1604459a46f2f5433ca4970 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:40.159575 19400 consensus_queue.cc:260] T a9522b81d1604459a46f2f5433ca4970 P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:40.159847 19400 ts_tablet_manager.cc:1398] T a9522b81d1604459a46f2f5433ca4970 P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:40.160686 19665 tablet_bootstrap.cc:492] T a9522b81d1604459a46f2f5433ca4970 P 61b887f0d40d497badeca5604c1f0546: No bootstrap required, opened a new log
I20240503 13:56:40.160781 19665 ts_tablet_manager.cc:1367] T a9522b81d1604459a46f2f5433ca4970 P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240503 13:56:40.161201 19665 raft_consensus.cc:348] T a9522b81d1604459a46f2f5433ca4970 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:40.161285 19665 raft_consensus.cc:374] T a9522b81d1604459a46f2f5433ca4970 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:40.161312 19665 raft_consensus.cc:729] T a9522b81d1604459a46f2f5433ca4970 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:40.161414 19665 consensus_queue.cc:260] T a9522b81d1604459a46f2f5433ca4970 P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:40.161679 19665 ts_tablet_manager.cc:1398] T a9522b81d1604459a46f2f5433ca4970 P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:40.174324 13051 raft_consensus.cc:1261] T 66121c347e6c443dac7c5123826a7ae1 P 60ba3ffe3b7746469834cd91c84be14e [term 2 FOLLOWER]: Refusing update from remote peer 61b887f0d40d497badeca5604c1f0546: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 13:56:40.174816 19349 consensus_queue.cc:1035] T 66121c347e6c443dac7c5123826a7ae1 P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:40.186654 19349 consensus_queue.cc:1035] T fff6b70fd6094e2ea86b03ff34b2f09b P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:40.190678 19828 consensus_queue.cc:1035] T fff6b70fd6094e2ea86b03ff34b2f09b P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:40.254248 13314 raft_consensus.cc:1261] T 66121c347e6c443dac7c5123826a7ae1 P ba0194098aa84449ae17e8a0d4c43703 [term 2 FOLLOWER]: Refusing update from remote peer 61b887f0d40d497badeca5604c1f0546: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 13:56:40.254841 19828 consensus_queue.cc:1035] T 66121c347e6c443dac7c5123826a7ae1 P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:40.257498 13183 raft_consensus.cc:1261] T 66121c347e6c443dac7c5123826a7ae1 P 8ccf5a4ac450414c9d3579bd5800b770 [term 2 FOLLOWER]: Refusing update from remote peer 61b887f0d40d497badeca5604c1f0546: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 13:56:40.258168 19828 consensus_queue.cc:1035] T 66121c347e6c443dac7c5123826a7ae1 P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:40.274713 19691 consensus_queue.cc:1035] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:40.278558 19307 consensus_queue.cc:1035] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:40.322371 19402 raft_consensus.cc:482] T a9522b81d1604459a46f2f5433ca4970 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:40.322588 19402 raft_consensus.cc:504] T a9522b81d1604459a46f2f5433ca4970 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:40.322983 19402 leader_election.cc:288] T a9522b81d1604459a46f2f5433ca4970 P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153), 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951)
I20240503 13:56:40.323365 15624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9522b81d1604459a46f2f5433ca4970" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d" is_pre_election: true
I20240503 13:56:40.323382 15492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9522b81d1604459a46f2f5433ca4970" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546" is_pre_election: true
I20240503 13:56:40.323518 15492 raft_consensus.cc:2454] T a9522b81d1604459a46f2f5433ca4970 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 0.
I20240503 13:56:40.323518 15624 raft_consensus.cc:2454] T a9522b81d1604459a46f2f5433ca4970 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 0.
I20240503 13:56:40.323886 13383 leader_election.cc:302] T a9522b81d1604459a46f2f5433ca4970 P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61b887f0d40d497badeca5604c1f0546, 7690ee79577c45e3a2c8f8b552c0bff8; no voters:
I20240503 13:56:40.324111 19402 raft_consensus.cc:2781] T a9522b81d1604459a46f2f5433ca4970 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:40.324158 19402 raft_consensus.cc:482] T a9522b81d1604459a46f2f5433ca4970 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:40.324183 19402 raft_consensus.cc:3037] T a9522b81d1604459a46f2f5433ca4970 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:40.325176 19402 raft_consensus.cc:504] T a9522b81d1604459a46f2f5433ca4970 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:40.325520 19402 leader_election.cc:288] T a9522b81d1604459a46f2f5433ca4970 P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 election: Requested vote from peers 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153), 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951)
I20240503 13:56:40.325816 15492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9522b81d1604459a46f2f5433ca4970" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546"
I20240503 13:56:40.325937 15492 raft_consensus.cc:3037] T a9522b81d1604459a46f2f5433ca4970 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:40.326969 15624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9522b81d1604459a46f2f5433ca4970" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d"
I20240503 13:56:40.327085 15624 raft_consensus.cc:3037] T a9522b81d1604459a46f2f5433ca4970 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:40.327118 15492 raft_consensus.cc:2454] T a9522b81d1604459a46f2f5433ca4970 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 1.
I20240503 13:56:40.327400 13383 leader_election.cc:302] T a9522b81d1604459a46f2f5433ca4970 P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61b887f0d40d497badeca5604c1f0546, 7690ee79577c45e3a2c8f8b552c0bff8; no voters:
I20240503 13:56:40.327558 19402 raft_consensus.cc:2781] T a9522b81d1604459a46f2f5433ca4970 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:40.327708 19402 raft_consensus.cc:686] T a9522b81d1604459a46f2f5433ca4970 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 LEADER]: Becoming Leader. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Running, Role: LEADER
I20240503 13:56:40.327847 19402 consensus_queue.cc:237] T a9522b81d1604459a46f2f5433ca4970 P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:40.328054 15624 raft_consensus.cc:2454] T a9522b81d1604459a46f2f5433ca4970 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 1.
I20240503 13:56:40.330250 12903 catalog_manager.cc:5484] T a9522b81d1604459a46f2f5433ca4970 P 7690ee79577c45e3a2c8f8b552c0bff8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68). New cstate: current_term: 1 leader_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:40.347834 15604 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 13:56:40.349409 13032 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:40.350564 13164 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:40.354666 15736 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:56:40.356153 13295 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 13:56:40.369048 15867 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 13:56:40.374109 19618 consensus_queue.cc:1035] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:40.377748 19618 consensus_queue.cc:1035] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:40.395696 15472 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 13:56:40.398308 13424 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:40.445657 13051 consensus_queue.cc:237] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:40.447052 15492 raft_consensus.cc:1261] T 38596c626067458495846931e0ecb7eb P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Refusing update from remote peer 60ba3ffe3b7746469834cd91c84be14e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:56:40.447062 15888 raft_consensus.cc:1261] T 38596c626067458495846931e0ecb7eb P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Refusing update from remote peer 60ba3ffe3b7746469834cd91c84be14e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:56:40.447435 19307 consensus_queue.cc:1035] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:56:40.447814 19307 consensus_queue.cc:1035] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:56:40.449286 19691 raft_consensus.cc:2932] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:40.450022 12903 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:56:40.450055 15889 raft_consensus.cc:2932] T 38596c626067458495846931e0ecb7eb P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:40.451982 15493 raft_consensus.cc:2932] T 38596c626067458495846931e0ecb7eb P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:40.452468 12912 catalog_manager.cc:5484] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e reported cstate change: config changed from index 2 to 3, VOTER 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71) evicted. New cstate: current_term: 1 leader_uuid: "60ba3ffe3b7746469834cd91c84be14e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:40.455925 15736 tablet_service.cc:1514] Processing DeleteTablet for tablet 38596c626067458495846931e0ecb7eb with delete_type TABLET_DATA_TOMBSTONED (TS 76c386bef61a4bd5a907266a5118ffe2 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:44984
I20240503 13:56:40.456743 20112 tablet_replica.cc:330] T 38596c626067458495846931e0ecb7eb P 76c386bef61a4bd5a907266a5118ffe2: stopping tablet replica
I20240503 13:56:40.456923 20112 raft_consensus.cc:2229] T 38596c626067458495846931e0ecb7eb P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:40.457098 20112 raft_consensus.cc:2258] T 38596c626067458495846931e0ecb7eb P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:40.457542 20112 ts_tablet_manager.cc:1875] T 38596c626067458495846931e0ecb7eb P 76c386bef61a4bd5a907266a5118ffe2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:56:40.458567 15624 raft_consensus.cc:1261] T a9522b81d1604459a46f2f5433ca4970 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Refusing update from remote peer 7690ee79577c45e3a2c8f8b552c0bff8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:40.459123 19402 consensus_queue.cc:1035] T a9522b81d1604459a46f2f5433ca4970 P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:40.459879 15493 raft_consensus.cc:1261] T a9522b81d1604459a46f2f5433ca4970 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Refusing update from remote peer 7690ee79577c45e3a2c8f8b552c0bff8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:40.460284 20112 ts_tablet_manager.cc:1888] T 38596c626067458495846931e0ecb7eb P 76c386bef61a4bd5a907266a5118ffe2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 13:56:40.460325 19402 consensus_queue.cc:1035] T a9522b81d1604459a46f2f5433ca4970 P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:40.460407 20112 log.cc:1196] T 38596c626067458495846931e0ecb7eb P 76c386bef61a4bd5a907266a5118ffe2: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/wal/wals/38596c626067458495846931e0ecb7eb
I20240503 13:56:40.461056 12899 catalog_manager.cc:4830] TS 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885): tablet 38596c626067458495846931e0ecb7eb (table rebalancer_extra_table_4 [id=51ad326dfca444208bdd2a00d9582876]) successfully deleted
I20240503 13:56:40.462545 12903 catalog_manager.cc:5576] TS 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951) does not have the latest schema for tablet a9522b81d1604459a46f2f5433ca4970 (table rebalancer_dynamic_table [id=3d3c742fef2046d6861c48cf78324535]). Expected version 1 got 0
I20240503 13:56:40.463764 12903 catalog_manager.cc:5576] TS 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153) does not have the latest schema for tablet a9522b81d1604459a46f2f5433ca4970 (table rebalancer_dynamic_table [id=3d3c742fef2046d6861c48cf78324535]). Expected version 1 got 0
I20240503 13:56:40.465518 20122 tablet.cc:1722] T a9522b81d1604459a46f2f5433ca4970 P 009f849ee7b54c80a3f5264695047e4d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:40.466310 12903 catalog_manager.cc:5576] TS 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153) does not have the latest schema for tablet a9522b81d1604459a46f2f5433ca4970 (table rebalancer_dynamic_table [id=3d3c742fef2046d6861c48cf78324535]). Expected version 1 got 0
I20240503 13:56:40.466811 20122 tablet.cc:1620] T a9522b81d1604459a46f2f5433ca4970 P 009f849ee7b54c80a3f5264695047e4d: MemRowSet was empty: no flush needed.
I20240503 13:56:40.466802 12906 catalog_manager.cc:5576] TS 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951) does not have the latest schema for tablet a9522b81d1604459a46f2f5433ca4970 (table rebalancer_dynamic_table [id=3d3c742fef2046d6861c48cf78324535]). Expected version 1 got 0
I20240503 13:56:40.467492 20123 tablet.cc:1722] T a9522b81d1604459a46f2f5433ca4970 P 61b887f0d40d497badeca5604c1f0546: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:40.467979 20123 tablet.cc:1620] T a9522b81d1604459a46f2f5433ca4970 P 61b887f0d40d497badeca5604c1f0546: MemRowSet was empty: no flush needed.
I20240503 13:56:40.468362 20121 tablet.cc:1722] T a9522b81d1604459a46f2f5433ca4970 P 7690ee79577c45e3a2c8f8b552c0bff8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:40.468703 20121 tablet.cc:1620] T a9522b81d1604459a46f2f5433ca4970 P 7690ee79577c45e3a2c8f8b552c0bff8: MemRowSet was empty: no flush needed.
I20240503 13:56:40.475472 12911 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_extra_table_30"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:56:40.475562 12906 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=3d3c742fef2046d6861c48cf78324535] alter complete (version 1)
I20240503 13:56:40.482677 15736 tablet_service.cc:1467] Processing CreateTablet for tablet bff36bb937a149c2890467aed1e54bca (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=eeedfc2be8b34e119e78afe55332d1fd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:40.483031 15736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bff36bb937a149c2890467aed1e54bca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:40.483204 13165 tablet_service.cc:1467] Processing CreateTablet for tablet 3f959d7bf8b64d8d8737a2223b9798ea (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=eeedfc2be8b34e119e78afe55332d1fd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:40.483449 13165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f959d7bf8b64d8d8737a2223b9798ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:40.483422 15867 tablet_service.cc:1467] Processing CreateTablet for tablet bff36bb937a149c2890467aed1e54bca (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=eeedfc2be8b34e119e78afe55332d1fd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:40.483680 15867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bff36bb937a149c2890467aed1e54bca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:40.485843 20135 tablet_bootstrap.cc:492] T bff36bb937a149c2890467aed1e54bca P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:40.485827 13163 tablet_service.cc:1467] Processing CreateTablet for tablet e020e9ba80c34c2d90950f6570d1a0ab (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=eeedfc2be8b34e119e78afe55332d1fd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:40.486002 20134 tablet_bootstrap.cc:492] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:40.486083 13163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e020e9ba80c34c2d90950f6570d1a0ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:40.486064 15868 tablet_service.cc:1467] Processing CreateTablet for tablet 3f959d7bf8b64d8d8737a2223b9798ea (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=eeedfc2be8b34e119e78afe55332d1fd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:40.486279 15868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f959d7bf8b64d8d8737a2223b9798ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:40.486878 20135 tablet_bootstrap.cc:654] T bff36bb937a149c2890467aed1e54bca P e47105a0f46e4511b7112e571688a4c7: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:40.486989 20134 tablet_bootstrap.cc:654] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:40.487953 20135 tablet_bootstrap.cc:492] T bff36bb937a149c2890467aed1e54bca P e47105a0f46e4511b7112e571688a4c7: No bootstrap required, opened a new log
I20240503 13:56:40.488044 20135 ts_tablet_manager.cc:1367] T bff36bb937a149c2890467aed1e54bca P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:56:40.488094 20134 tablet_bootstrap.cc:492] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:40.488166 20134 ts_tablet_manager.cc:1367] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:56:40.488523 20135 raft_consensus.cc:348] T bff36bb937a149c2890467aed1e54bca P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:40.488641 20135 raft_consensus.cc:374] T bff36bb937a149c2890467aed1e54bca P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:40.488592 20134 raft_consensus.cc:348] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:40.488669 20135 raft_consensus.cc:729] T bff36bb937a149c2890467aed1e54bca P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:40.488679 20134 raft_consensus.cc:374] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:40.488709 20134 raft_consensus.cc:729] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:40.488721 13295 tablet_service.cc:1467] Processing CreateTablet for tablet 3f959d7bf8b64d8d8737a2223b9798ea (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=eeedfc2be8b34e119e78afe55332d1fd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:40.488776 20135 consensus_queue.cc:260] T bff36bb937a149c2890467aed1e54bca P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:40.488811 20134 consensus_queue.cc:260] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:40.489005 13295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f959d7bf8b64d8d8737a2223b9798ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:40.489076 20134 ts_tablet_manager.cc:1398] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:40.489084 20135 ts_tablet_manager.cc:1398] T bff36bb937a149c2890467aed1e54bca P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:40.489244 20135 tablet_bootstrap.cc:492] T 3f959d7bf8b64d8d8737a2223b9798ea P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:40.490132 20135 tablet_bootstrap.cc:654] T 3f959d7bf8b64d8d8737a2223b9798ea P e47105a0f46e4511b7112e571688a4c7: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:40.485827 13164 tablet_service.cc:1467] Processing CreateTablet for tablet bff36bb937a149c2890467aed1e54bca (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=eeedfc2be8b34e119e78afe55332d1fd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:40.491149 13164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bff36bb937a149c2890467aed1e54bca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:40.491401 13296 tablet_service.cc:1467] Processing CreateTablet for tablet e020e9ba80c34c2d90950f6570d1a0ab (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=eeedfc2be8b34e119e78afe55332d1fd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:40.491624 13296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e020e9ba80c34c2d90950f6570d1a0ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:40.491990 20135 tablet_bootstrap.cc:492] T 3f959d7bf8b64d8d8737a2223b9798ea P e47105a0f46e4511b7112e571688a4c7: No bootstrap required, opened a new log
I20240503 13:56:40.492084 20135 ts_tablet_manager.cc:1367] T 3f959d7bf8b64d8d8737a2223b9798ea P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:56:40.492503 20135 raft_consensus.cc:348] T 3f959d7bf8b64d8d8737a2223b9798ea P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:40.492612 20135 raft_consensus.cc:374] T 3f959d7bf8b64d8d8737a2223b9798ea P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:40.492650 20135 raft_consensus.cc:729] T 3f959d7bf8b64d8d8737a2223b9798ea P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:40.492762 20135 consensus_queue.cc:260] T 3f959d7bf8b64d8d8737a2223b9798ea P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:40.493034 20135 ts_tablet_manager.cc:1398] T 3f959d7bf8b64d8d8737a2223b9798ea P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:40.497515 15868 tablet_service.cc:1467] Processing CreateTablet for tablet e020e9ba80c34c2d90950f6570d1a0ab (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=eeedfc2be8b34e119e78afe55332d1fd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:40.497802 15868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e020e9ba80c34c2d90950f6570d1a0ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:40.499459 20135 tablet_bootstrap.cc:492] T e020e9ba80c34c2d90950f6570d1a0ab P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:40.499594 20140 tablet_bootstrap.cc:492] T 3f959d7bf8b64d8d8737a2223b9798ea P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:40.500555 20140 tablet_bootstrap.cc:654] T 3f959d7bf8b64d8d8737a2223b9798ea P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:40.500615 20135 tablet_bootstrap.cc:654] T e020e9ba80c34c2d90950f6570d1a0ab P e47105a0f46e4511b7112e571688a4c7: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:40.501271 20136 tablet_bootstrap.cc:492] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:40.502173 20136 tablet_bootstrap.cc:654] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:40.502992 20140 tablet_bootstrap.cc:492] T 3f959d7bf8b64d8d8737a2223b9798ea P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:40.503074 20140 ts_tablet_manager.cc:1367] T 3f959d7bf8b64d8d8737a2223b9798ea P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 13:56:40.503216 20136 tablet_bootstrap.cc:492] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:40.503302 20136 ts_tablet_manager.cc:1367] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:40.503504 20140 raft_consensus.cc:348] T 3f959d7bf8b64d8d8737a2223b9798ea P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:40.503607 20140 raft_consensus.cc:374] T 3f959d7bf8b64d8d8737a2223b9798ea P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:40.503628 20140 raft_consensus.cc:729] T 3f959d7bf8b64d8d8737a2223b9798ea P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:40.503692 20136 raft_consensus.cc:348] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:40.503799 20136 raft_consensus.cc:374] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:40.503772 20140 consensus_queue.cc:260] T 3f959d7bf8b64d8d8737a2223b9798ea P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:40.503839 20136 raft_consensus.cc:729] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:40.503971 20136 consensus_queue.cc:260] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:40.504032 20140 ts_tablet_manager.cc:1398] T 3f959d7bf8b64d8d8737a2223b9798ea P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:40.504171 20140 tablet_bootstrap.cc:492] T e020e9ba80c34c2d90950f6570d1a0ab P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:40.504324 20136 ts_tablet_manager.cc:1398] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:40.505692 20136 tablet_bootstrap.cc:492] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:40.507472 20136 tablet_bootstrap.cc:654] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:40.507898 20140 tablet_bootstrap.cc:654] T e020e9ba80c34c2d90950f6570d1a0ab P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:40.508795 20136 tablet_bootstrap.cc:492] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:40.508929 20136 ts_tablet_manager.cc:1367] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:56:40.509527 20136 raft_consensus.cc:348] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:40.509944 20136 raft_consensus.cc:374] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:40.510040 20136 raft_consensus.cc:729] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:40.510411 20136 consensus_queue.cc:260] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:40.511269 20136 ts_tablet_manager.cc:1398] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:56:40.511482 20136 tablet_bootstrap.cc:492] T bff36bb937a149c2890467aed1e54bca P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:40.511600 20135 tablet_bootstrap.cc:492] T e020e9ba80c34c2d90950f6570d1a0ab P e47105a0f46e4511b7112e571688a4c7: No bootstrap required, opened a new log
I20240503 13:56:40.511713 20135 ts_tablet_manager.cc:1367] T e020e9ba80c34c2d90950f6570d1a0ab P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.001s
I20240503 13:56:40.512197 20135 raft_consensus.cc:348] T e020e9ba80c34c2d90950f6570d1a0ab P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:40.512316 20135 raft_consensus.cc:374] T e020e9ba80c34c2d90950f6570d1a0ab P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:40.512356 20135 raft_consensus.cc:729] T e020e9ba80c34c2d90950f6570d1a0ab P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:40.512507 20135 consensus_queue.cc:260] T e020e9ba80c34c2d90950f6570d1a0ab P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:40.512804 20135 ts_tablet_manager.cc:1398] T e020e9ba80c34c2d90950f6570d1a0ab P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:40.513038 15624 consensus_queue.cc:237] T 3f5d536cffe742e69f4b0d3b93caf76c P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:40.513527 20140 tablet_bootstrap.cc:492] T e020e9ba80c34c2d90950f6570d1a0ab P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:40.513581 20136 tablet_bootstrap.cc:654] T bff36bb937a149c2890467aed1e54bca P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:40.513669 20140 ts_tablet_manager.cc:1367] T e020e9ba80c34c2d90950f6570d1a0ab P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.002s
I20240503 13:56:40.514673 20136 tablet_bootstrap.cc:492] T bff36bb937a149c2890467aed1e54bca P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:40.514688 13051 raft_consensus.cc:1261] T 3f5d536cffe742e69f4b0d3b93caf76c P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Refusing update from remote peer 009f849ee7b54c80a3f5264695047e4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:40.514756 20136 ts_tablet_manager.cc:1367] T bff36bb937a149c2890467aed1e54bca P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 13:56:40.515086 19586 consensus_queue.cc:1035] T 3f5d536cffe742e69f4b0d3b93caf76c P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:56:40.515161 20136 raft_consensus.cc:348] T bff36bb937a149c2890467aed1e54bca P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:40.515254 20136 raft_consensus.cc:374] T bff36bb937a149c2890467aed1e54bca P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:40.515275 20136 raft_consensus.cc:729] T bff36bb937a149c2890467aed1e54bca P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
W20240503 13:56:40.515316 15561 consensus_peers.cc:485] T 3f5d536cffe742e69f4b0d3b93caf76c P 009f849ee7b54c80a3f5264695047e4d -> Peer ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279): Couldn't send request to peer ba0194098aa84449ae17e8a0d4c43703. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3f5d536cffe742e69f4b0d3b93caf76c. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:40.515496 20136 consensus_queue.cc:260] T bff36bb937a149c2890467aed1e54bca P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:40.515766 20136 ts_tablet_manager.cc:1398] T bff36bb937a149c2890467aed1e54bca P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:40.515949 20140 raft_consensus.cc:348] T e020e9ba80c34c2d90950f6570d1a0ab P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:40.516093 20140 raft_consensus.cc:374] T e020e9ba80c34c2d90950f6570d1a0ab P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:40.516126 20140 raft_consensus.cc:729] T e020e9ba80c34c2d90950f6570d1a0ab P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:40.516239 20140 consensus_queue.cc:260] T e020e9ba80c34c2d90950f6570d1a0ab P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:40.516623 20140 ts_tablet_manager.cc:1398] T e020e9ba80c34c2d90950f6570d1a0ab P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 13:56:40.517268 15758 raft_consensus.cc:1261] T 3f5d536cffe742e69f4b0d3b93caf76c P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Refusing update from remote peer 009f849ee7b54c80a3f5264695047e4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:40.518182 19586 consensus_queue.cc:1035] T 3f5d536cffe742e69f4b0d3b93caf76c P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:56:40.520184 20152 raft_consensus.cc:2932] T 3f5d536cffe742e69f4b0d3b93caf76c P 009f849ee7b54c80a3f5264695047e4d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:40.520651 15758 raft_consensus.cc:2932] T 3f5d536cffe742e69f4b0d3b93caf76c P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:40.522536 13051 raft_consensus.cc:2932] T 3f5d536cffe742e69f4b0d3b93caf76c P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:40.524015 12912 catalog_manager.cc:5484] T 3f5d536cffe742e69f4b0d3b93caf76c P 76c386bef61a4bd5a907266a5118ffe2 reported cstate change: config changed from index -1 to 2, VOTER ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) added. New cstate: current_term: 1 leader_uuid: "009f849ee7b54c80a3f5264695047e4d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:40.545933 15624 consensus_queue.cc:237] T c8a53b7095e948f988c9151882bfa1bd P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:40.547605 15493 raft_consensus.cc:1261] T c8a53b7095e948f988c9151882bfa1bd P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Refusing update from remote peer 009f849ee7b54c80a3f5264695047e4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 13:56:40.547814 15561 consensus_peers.cc:485] T c8a53b7095e948f988c9151882bfa1bd P 009f849ee7b54c80a3f5264695047e4d -> Peer 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387): Couldn't send request to peer 8ccf5a4ac450414c9d3579bd5800b770. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c8a53b7095e948f988c9151882bfa1bd. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:40.548331 19586 consensus_queue.cc:1035] T c8a53b7095e948f988c9151882bfa1bd P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:56:40.552902 13314 raft_consensus.cc:1261] T c8a53b7095e948f988c9151882bfa1bd P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Refusing update from remote peer 009f849ee7b54c80a3f5264695047e4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:40.553324 19586 consensus_queue.cc:1035] T c8a53b7095e948f988c9151882bfa1bd P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:56:40.555073 20152 raft_consensus.cc:2932] T c8a53b7095e948f988c9151882bfa1bd P 009f849ee7b54c80a3f5264695047e4d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } }
I20240503 13:56:40.555495 13314 raft_consensus.cc:2932] T c8a53b7095e948f988c9151882bfa1bd P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } }
I20240503 13:56:40.558115 12911 catalog_manager.cc:5484] T c8a53b7095e948f988c9151882bfa1bd P 009f849ee7b54c80a3f5264695047e4d reported cstate change: config changed from index -1 to 2, VOTER 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66) added. New cstate: current_term: 1 leader_uuid: "009f849ee7b54c80a3f5264695047e4d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } }
I20240503 13:56:40.559751 15493 raft_consensus.cc:2932] T c8a53b7095e948f988c9151882bfa1bd P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } }
I20240503 13:56:40.581218 12903 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:56:40.586865 20121 tablet.cc:1722] T a9522b81d1604459a46f2f5433ca4970 P 7690ee79577c45e3a2c8f8b552c0bff8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:40.587209 20121 tablet.cc:1620] T a9522b81d1604459a46f2f5433ca4970 P 7690ee79577c45e3a2c8f8b552c0bff8: MemRowSet was empty: no flush needed.
I20240503 13:56:40.587466 20122 tablet.cc:1722] T a9522b81d1604459a46f2f5433ca4970 P 009f849ee7b54c80a3f5264695047e4d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:40.587466 20123 tablet.cc:1722] T a9522b81d1604459a46f2f5433ca4970 P 61b887f0d40d497badeca5604c1f0546: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:40.587733 20123 tablet.cc:1620] T a9522b81d1604459a46f2f5433ca4970 P 61b887f0d40d497badeca5604c1f0546: MemRowSet was empty: no flush needed.
I20240503 13:56:40.587733 20122 tablet.cc:1620] T a9522b81d1604459a46f2f5433ca4970 P 009f849ee7b54c80a3f5264695047e4d: MemRowSet was empty: no flush needed.
I20240503 13:56:40.589373 13445 consensus_queue.cc:237] T b1395d66d8e04ecdafd7a30c50aa1c9a P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:40.590790 13314 raft_consensus.cc:1261] T b1395d66d8e04ecdafd7a30c50aa1c9a P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Refusing update from remote peer 7690ee79577c45e3a2c8f8b552c0bff8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:56:40.591279 13183 raft_consensus.cc:1261] T b1395d66d8e04ecdafd7a30c50aa1c9a P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Refusing update from remote peer 7690ee79577c45e3a2c8f8b552c0bff8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:56:40.591338 20076 consensus_queue.cc:1035] T b1395d66d8e04ecdafd7a30c50aa1c9a P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:40.591961 19841 raft_consensus.cc:482] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:40.592069 19841 raft_consensus.cc:504] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:40.592470 19841 leader_election.cc:288] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:40.592819 13183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bff36bb937a149c2890467aed1e54bca" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770" is_pre_election: true
I20240503 13:56:40.592957 13183 raft_consensus.cc:2454] T bff36bb937a149c2890467aed1e54bca P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 0.
I20240503 13:56:40.593253 15694 leader_election.cc:302] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76c386bef61a4bd5a907266a5118ffe2, 8ccf5a4ac450414c9d3579bd5800b770; no voters:
I20240503 13:56:40.593421 19841 raft_consensus.cc:2781] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:40.593469 19841 raft_consensus.cc:482] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:40.593502 19841 raft_consensus.cc:3037] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:40.593480 15889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bff36bb937a149c2890467aed1e54bca" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7" is_pre_election: true
I20240503 13:56:40.593605 15889 raft_consensus.cc:2454] T bff36bb937a149c2890467aed1e54bca P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 0.
I20240503 13:56:40.594116 12903 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=3d3c742fef2046d6861c48cf78324535] alter complete (version 2)
I20240503 13:56:40.594445 19402 consensus_queue.cc:1035] T b1395d66d8e04ecdafd7a30c50aa1c9a P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:40.594545 19841 raft_consensus.cc:504] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:40.594902 19841 leader_election.cc:288] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 election: Requested vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:40.595332 15889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bff36bb937a149c2890467aed1e54bca" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7"
I20240503 13:56:40.595443 15889 raft_consensus.cc:3037] T bff36bb937a149c2890467aed1e54bca P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Advancing to term 1
W20240503 13:56:40.595589 13380 consensus_peers.cc:485] T b1395d66d8e04ecdafd7a30c50aa1c9a P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951): Couldn't send request to peer 009f849ee7b54c80a3f5264695047e4d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b1395d66d8e04ecdafd7a30c50aa1c9a. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:40.595901 13185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bff36bb937a149c2890467aed1e54bca" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770"
I20240503 13:56:40.595999 13185 raft_consensus.cc:3037] T bff36bb937a149c2890467aed1e54bca P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:40.596405 15889 raft_consensus.cc:2454] T bff36bb937a149c2890467aed1e54bca P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 1.
I20240503 13:56:40.596690 15691 leader_election.cc:302] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76c386bef61a4bd5a907266a5118ffe2, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:40.596966 13185 raft_consensus.cc:2454] T bff36bb937a149c2890467aed1e54bca P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 1.
I20240503 13:56:40.597592 19841 raft_consensus.cc:2781] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:40.597685 19841 raft_consensus.cc:686] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 [term 1 LEADER]: Becoming Leader. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Running, Role: LEADER
I20240503 13:56:40.597638 19402 raft_consensus.cc:2932] T b1395d66d8e04ecdafd7a30c50aa1c9a P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 009f849ee7b54c80a3f5264695047e4d (127.6.98.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:40.597807 19841 consensus_queue.cc:237] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:40.598929 13185 raft_consensus.cc:2932] T b1395d66d8e04ecdafd7a30c50aa1c9a P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 009f849ee7b54c80a3f5264695047e4d (127.6.98.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:40.599874 12912 catalog_manager.cc:5484] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71). New cstate: current_term: 1 leader_uuid: "76c386bef61a4bd5a907266a5118ffe2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } }
I20240503 13:56:40.600538 12911 catalog_manager.cc:5484] T b1395d66d8e04ecdafd7a30c50aa1c9a P 7690ee79577c45e3a2c8f8b552c0bff8 reported cstate change: config changed from index -1 to 4, VOTER 009f849ee7b54c80a3f5264695047e4d (127.6.98.70) added. New cstate: current_term: 1 leader_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:40.601104 13314 raft_consensus.cc:2932] T b1395d66d8e04ecdafd7a30c50aa1c9a P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 009f849ee7b54c80a3f5264695047e4d (127.6.98.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:40.603097 19019 raft_consensus.cc:482] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:40.603188 19019 raft_consensus.cc:504] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:40.603544 19019 leader_election.cc:288] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:40.604081 15888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e020e9ba80c34c2d90950f6570d1a0ab" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7" is_pre_election: true
I20240503 13:56:40.604204 15888 raft_consensus.cc:2454] T e020e9ba80c34c2d90950f6570d1a0ab P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 0.
I20240503 13:56:40.604411 13314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e020e9ba80c34c2d90950f6570d1a0ab" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703" is_pre_election: true
I20240503 13:56:40.604547 13314 raft_consensus.cc:2454] T e020e9ba80c34c2d90950f6570d1a0ab P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 0.
I20240503 13:56:40.604573 13118 leader_election.cc:302] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ccf5a4ac450414c9d3579bd5800b770, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:40.604827 19019 raft_consensus.cc:2781] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:40.604887 19019 raft_consensus.cc:482] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:40.604931 19019 raft_consensus.cc:3037] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:40.605933 19019 raft_consensus.cc:504] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:40.606261 19019 leader_election.cc:288] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 election: Requested vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:40.607082 15888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e020e9ba80c34c2d90950f6570d1a0ab" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7"
I20240503 13:56:40.607192 15888 raft_consensus.cc:3037] T e020e9ba80c34c2d90950f6570d1a0ab P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:40.607225 13314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e020e9ba80c34c2d90950f6570d1a0ab" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703"
I20240503 13:56:40.607323 13314 raft_consensus.cc:3037] T e020e9ba80c34c2d90950f6570d1a0ab P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:40.608093 15888 raft_consensus.cc:2454] T e020e9ba80c34c2d90950f6570d1a0ab P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 1.
I20240503 13:56:40.608248 13314 raft_consensus.cc:2454] T e020e9ba80c34c2d90950f6570d1a0ab P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 1.
I20240503 13:56:40.608428 13118 leader_election.cc:302] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ccf5a4ac450414c9d3579bd5800b770, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:40.608603 19019 raft_consensus.cc:2781] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:40.608685 19019 raft_consensus.cc:686] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 LEADER]: Becoming Leader. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Running, Role: LEADER
I20240503 13:56:40.608820 19019 consensus_queue.cc:237] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:40.612416 12903 catalog_manager.cc:5484] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66). New cstate: current_term: 1 leader_uuid: "8ccf5a4ac450414c9d3579bd5800b770" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:40.624821 13314 consensus_queue.cc:237] T 258d1ec1f0694c15a4d41bd064f35b4e P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:40.625871 19998 raft_consensus.cc:482] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:40.625975 19998 raft_consensus.cc:504] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:40.626331 13445 raft_consensus.cc:1261] T 258d1ec1f0694c15a4d41bd064f35b4e P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:56:40.626384 19998 leader_election.cc:288] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:40.626645 13185 raft_consensus.cc:1261] T 258d1ec1f0694c15a4d41bd064f35b4e P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:56:40.626865 18873 consensus_queue.cc:1035] T 258d1ec1f0694c15a4d41bd064f35b4e P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:40.626951 13314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f959d7bf8b64d8d8737a2223b9798ea" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703" is_pre_election: true
I20240503 13:56:40.626967 15888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f959d7bf8b64d8d8737a2223b9798ea" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7" is_pre_election: true
I20240503 13:56:40.627035 13314 raft_consensus.cc:2454] T 3f959d7bf8b64d8d8737a2223b9798ea P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 0.
I20240503 13:56:40.627089 15888 raft_consensus.cc:2454] T 3f959d7bf8b64d8d8737a2223b9798ea P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 0.
I20240503 13:56:40.627257 13121 leader_election.cc:302] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ccf5a4ac450414c9d3579bd5800b770, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:40.627490 19998 raft_consensus.cc:2781] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:40.627542 19998 raft_consensus.cc:482] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:40.627566 19998 raft_consensus.cc:3037] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:40.628423 19998 raft_consensus.cc:504] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:40.628757 19998 leader_election.cc:288] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 election: Requested vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:40.629079 15888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f959d7bf8b64d8d8737a2223b9798ea" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7"
I20240503 13:56:40.629186 15888 raft_consensus.cc:3037] T 3f959d7bf8b64d8d8737a2223b9798ea P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:40.629194 13314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f959d7bf8b64d8d8737a2223b9798ea" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703"
I20240503 13:56:40.629314 13314 raft_consensus.cc:3037] T 3f959d7bf8b64d8d8737a2223b9798ea P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
W20240503 13:56:40.629727 13252 consensus_peers.cc:485] T 258d1ec1f0694c15a4d41bd064f35b4e P ba0194098aa84449ae17e8a0d4c43703 -> Peer 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153): Couldn't send request to peer 61b887f0d40d497badeca5604c1f0546. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 258d1ec1f0694c15a4d41bd064f35b4e. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:40.630041 18873 consensus_queue.cc:1035] T 258d1ec1f0694c15a4d41bd064f35b4e P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:40.630160 15888 raft_consensus.cc:2454] T 3f959d7bf8b64d8d8737a2223b9798ea P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 1.
I20240503 13:56:40.630446 13314 raft_consensus.cc:2454] T 3f959d7bf8b64d8d8737a2223b9798ea P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 1.
I20240503 13:56:40.630510 13118 leader_election.cc:302] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ccf5a4ac450414c9d3579bd5800b770, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:40.630654 19998 raft_consensus.cc:2781] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:40.630734 19998 raft_consensus.cc:686] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 LEADER]: Becoming Leader. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Running, Role: LEADER
I20240503 13:56:40.630863 19998 consensus_queue.cc:237] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:40.632272 18914 raft_consensus.cc:2932] T 258d1ec1f0694c15a4d41bd064f35b4e P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 61b887f0d40d497badeca5604c1f0546 (127.6.98.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:40.632712 13185 raft_consensus.cc:2932] T 258d1ec1f0694c15a4d41bd064f35b4e P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 61b887f0d40d497badeca5604c1f0546 (127.6.98.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:40.634565 13445 raft_consensus.cc:2932] T 258d1ec1f0694c15a4d41bd064f35b4e P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 61b887f0d40d497badeca5604c1f0546 (127.6.98.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:40.635432 12903 catalog_manager.cc:5484] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66). New cstate: current_term: 1 leader_uuid: "8ccf5a4ac450414c9d3579bd5800b770" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:40.637351 12906 catalog_manager.cc:5484] T 258d1ec1f0694c15a4d41bd064f35b4e P 7690ee79577c45e3a2c8f8b552c0bff8 reported cstate change: config changed from index -1 to 4, VOTER 61b887f0d40d497badeca5604c1f0546 (127.6.98.69) added. New cstate: current_term: 1 leader_uuid: "ba0194098aa84449ae17e8a0d4c43703" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:40.657173 15888 consensus_queue.cc:237] T eb92a057165f4da7b96d910d847ff761 P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:40.658758 13185 raft_consensus.cc:1261] T eb92a057165f4da7b96d910d847ff761 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Refusing update from remote peer e47105a0f46e4511b7112e571688a4c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 13:56:40.659054 15825 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P e47105a0f46e4511b7112e571688a4c7 -> Peer 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885): Couldn't send request to peer 76c386bef61a4bd5a907266a5118ffe2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eb92a057165f4da7b96d910d847ff761. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:40.659283 15493 raft_consensus.cc:1261] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Refusing update from remote peer e47105a0f46e4511b7112e571688a4c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:40.659262 20232 consensus_queue.cc:1035] T eb92a057165f4da7b96d910d847ff761 P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:56:40.659628 20232 consensus_queue.cc:1035] T eb92a057165f4da7b96d910d847ff761 P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:56:40.661375 19351 raft_consensus.cc:2932] T eb92a057165f4da7b96d910d847ff761 P e47105a0f46e4511b7112e571688a4c7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:40.661777 15493 raft_consensus.cc:2932] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:40.663944 13185 raft_consensus.cc:2932] T eb92a057165f4da7b96d910d847ff761 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:40.664397 12911 catalog_manager.cc:5484] T eb92a057165f4da7b96d910d847ff761 P e47105a0f46e4511b7112e571688a4c7 reported cstate change: config changed from index -1 to 2, VOTER 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71) added. New cstate: current_term: 1 leader_uuid: "e47105a0f46e4511b7112e571688a4c7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:40.697670 12906 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:56:40.703866 20121 tablet.cc:1722] T a9522b81d1604459a46f2f5433ca4970 P 7690ee79577c45e3a2c8f8b552c0bff8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:40.704319 20123 tablet.cc:1722] T a9522b81d1604459a46f2f5433ca4970 P 61b887f0d40d497badeca5604c1f0546: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:40.704746 20122 tablet.cc:1722] T a9522b81d1604459a46f2f5433ca4970 P 009f849ee7b54c80a3f5264695047e4d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:40.709504 12903 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table_new_name [id=3d3c742fef2046d6861c48cf78324535] alter complete (version 3)
I20240503 13:56:40.763396 13032 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:40.770865 15736 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 13:56:40.771940 15472 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 13:56:40.779672 13424 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:40.779795 15604 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 13:56:40.800304 13164 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 13:56:40.812153 13296 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 13:56:40.812789 12907 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:40.812932 12907 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:40.816545 12907 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Sending DeleteTablet for 3 replicas of tablet a9522b81d1604459a46f2f5433ca4970
I20240503 13:56:40.817294 13424 tablet_service.cc:1514] Processing DeleteTablet for tablet a9522b81d1604459a46f2f5433ca4970 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:40 UTC) from {username='slave'} at 127.0.0.1:33678
I20240503 13:56:40.817348 15472 tablet_service.cc:1514] Processing DeleteTablet for tablet a9522b81d1604459a46f2f5433ca4970 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:40 UTC) from {username='slave'} at 127.0.0.1:58758
I20240503 13:56:40.817766 15868 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 13:56:40.818718 20288 tablet_replica.cc:330] T a9522b81d1604459a46f2f5433ca4970 P 61b887f0d40d497badeca5604c1f0546: stopping tablet replica
I20240503 13:56:40.818904 20288 raft_consensus.cc:2229] T a9522b81d1604459a46f2f5433ca4970 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:40.819038 20288 raft_consensus.cc:2258] T a9522b81d1604459a46f2f5433ca4970 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:40.819947 20288 ts_tablet_manager.cc:1875] T a9522b81d1604459a46f2f5433ca4970 P 61b887f0d40d497badeca5604c1f0546: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:40.822455 20288 ts_tablet_manager.cc:1888] T a9522b81d1604459a46f2f5433ca4970 P 61b887f0d40d497badeca5604c1f0546: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:40.822623 20288 log.cc:1196] T a9522b81d1604459a46f2f5433ca4970 P 61b887f0d40d497badeca5604c1f0546: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/wal/wals/a9522b81d1604459a46f2f5433ca4970
I20240503 13:56:40.822935 20288 ts_tablet_manager.cc:1909] T a9522b81d1604459a46f2f5433ca4970 P 61b887f0d40d497badeca5604c1f0546: Deleting consensus metadata
I20240503 13:56:40.824666 15604 tablet_service.cc:1514] Processing DeleteTablet for tablet a9522b81d1604459a46f2f5433ca4970 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:40 UTC) from {username='slave'} at 127.0.0.1:60718
I20240503 13:56:40.825145 12899 catalog_manager.cc:4830] TS 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153): tablet a9522b81d1604459a46f2f5433ca4970 (table rebalancer_dynamic_table_new_name [id=3d3c742fef2046d6861c48cf78324535]) successfully deleted
I20240503 13:56:40.829809 20289 tablet_replica.cc:330] T a9522b81d1604459a46f2f5433ca4970 P 009f849ee7b54c80a3f5264695047e4d: stopping tablet replica
I20240503 13:56:40.830031 20289 raft_consensus.cc:2229] T a9522b81d1604459a46f2f5433ca4970 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:40.830217 20289 raft_consensus.cc:2258] T a9522b81d1604459a46f2f5433ca4970 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:40.830847 20289 ts_tablet_manager.cc:1875] T a9522b81d1604459a46f2f5433ca4970 P 009f849ee7b54c80a3f5264695047e4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:40.832072 20287 tablet_replica.cc:330] T a9522b81d1604459a46f2f5433ca4970 P 7690ee79577c45e3a2c8f8b552c0bff8: stopping tablet replica
I20240503 13:56:40.832270 20287 raft_consensus.cc:2229] T a9522b81d1604459a46f2f5433ca4970 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:56:40.832566 20287 raft_consensus.cc:2258] T a9522b81d1604459a46f2f5433ca4970 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:40.833581 20289 ts_tablet_manager.cc:1888] T a9522b81d1604459a46f2f5433ca4970 P 009f849ee7b54c80a3f5264695047e4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:40.833704 20289 log.cc:1196] T a9522b81d1604459a46f2f5433ca4970 P 009f849ee7b54c80a3f5264695047e4d: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/wal/wals/a9522b81d1604459a46f2f5433ca4970
I20240503 13:56:40.834009 20289 ts_tablet_manager.cc:1909] T a9522b81d1604459a46f2f5433ca4970 P 009f849ee7b54c80a3f5264695047e4d: Deleting consensus metadata
I20240503 13:56:40.833843 20287 ts_tablet_manager.cc:1875] T a9522b81d1604459a46f2f5433ca4970 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:40.834810 12896 catalog_manager.cc:4830] TS 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951): tablet a9522b81d1604459a46f2f5433ca4970 (table rebalancer_dynamic_table_new_name [id=3d3c742fef2046d6861c48cf78324535]) successfully deleted
I20240503 13:56:40.836839 20287 ts_tablet_manager.cc:1888] T a9522b81d1604459a46f2f5433ca4970 P 7690ee79577c45e3a2c8f8b552c0bff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:40.836902 20287 log.cc:1196] T a9522b81d1604459a46f2f5433ca4970 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal/wals/a9522b81d1604459a46f2f5433ca4970
I20240503 13:56:40.837142 20287 ts_tablet_manager.cc:1909] T a9522b81d1604459a46f2f5433ca4970 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting consensus metadata
I20240503 13:56:40.837771 12896 catalog_manager.cc:4830] TS 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109): tablet a9522b81d1604459a46f2f5433ca4970 (table rebalancer_dynamic_table_new_name [id=3d3c742fef2046d6861c48cf78324535]) successfully deleted
I20240503 13:56:40.918694 12907 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:56:40.924558 13032 tablet_service.cc:1467] Processing CreateTablet for tablet d5c7e23182354fb792aa175f6dcb5f20 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2a01aba75336445ca17e5ef317d14b36]), partition=
I20240503 13:56:40.924654 15868 tablet_service.cc:1467] Processing CreateTablet for tablet d5c7e23182354fb792aa175f6dcb5f20 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2a01aba75336445ca17e5ef317d14b36]), partition=
I20240503 13:56:40.924857 13032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5c7e23182354fb792aa175f6dcb5f20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:40.924901 15868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5c7e23182354fb792aa175f6dcb5f20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:40.926784 20135 tablet_bootstrap.cc:492] T d5c7e23182354fb792aa175f6dcb5f20 P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:40.927176 15472 tablet_service.cc:1467] Processing CreateTablet for tablet d5c7e23182354fb792aa175f6dcb5f20 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2a01aba75336445ca17e5ef317d14b36]), partition=
I20240503 13:56:40.927400 15472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5c7e23182354fb792aa175f6dcb5f20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:40.927707 20135 tablet_bootstrap.cc:654] T d5c7e23182354fb792aa175f6dcb5f20 P e47105a0f46e4511b7112e571688a4c7: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:40.928807 20135 tablet_bootstrap.cc:492] T d5c7e23182354fb792aa175f6dcb5f20 P e47105a0f46e4511b7112e571688a4c7: No bootstrap required, opened a new log
I20240503 13:56:40.928877 20135 ts_tablet_manager.cc:1367] T d5c7e23182354fb792aa175f6dcb5f20 P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 13:56:40.929231 20135 raft_consensus.cc:348] T d5c7e23182354fb792aa175f6dcb5f20 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:40.929313 20135 raft_consensus.cc:374] T d5c7e23182354fb792aa175f6dcb5f20 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:40.929342 20135 raft_consensus.cc:729] T d5c7e23182354fb792aa175f6dcb5f20 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:40.929452 20135 consensus_queue.cc:260] T d5c7e23182354fb792aa175f6dcb5f20 P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:40.929733 20333 tablet_bootstrap.cc:492] T d5c7e23182354fb792aa175f6dcb5f20 P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:40.929733 20135 ts_tablet_manager.cc:1398] T d5c7e23182354fb792aa175f6dcb5f20 P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:40.930310 20331 tablet_bootstrap.cc:492] T d5c7e23182354fb792aa175f6dcb5f20 P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:40.931152 20333 tablet_bootstrap.cc:654] T d5c7e23182354fb792aa175f6dcb5f20 P 61b887f0d40d497badeca5604c1f0546: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:40.932967 20331 tablet_bootstrap.cc:654] T d5c7e23182354fb792aa175f6dcb5f20 P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:40.939386 20333 tablet_bootstrap.cc:492] T d5c7e23182354fb792aa175f6dcb5f20 P 61b887f0d40d497badeca5604c1f0546: No bootstrap required, opened a new log
I20240503 13:56:40.939497 20333 ts_tablet_manager.cc:1367] T d5c7e23182354fb792aa175f6dcb5f20 P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 13:56:40.939997 20333 raft_consensus.cc:348] T d5c7e23182354fb792aa175f6dcb5f20 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:40.940147 20333 raft_consensus.cc:374] T d5c7e23182354fb792aa175f6dcb5f20 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:40.940872 20333 raft_consensus.cc:729] T d5c7e23182354fb792aa175f6dcb5f20 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:40.941040 20333 consensus_queue.cc:260] T d5c7e23182354fb792aa175f6dcb5f20 P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:40.942292 20331 tablet_bootstrap.cc:492] T d5c7e23182354fb792aa175f6dcb5f20 P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:40.942392 20331 ts_tablet_manager.cc:1367] T d5c7e23182354fb792aa175f6dcb5f20 P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 13:56:40.942806 20331 raft_consensus.cc:348] T d5c7e23182354fb792aa175f6dcb5f20 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:40.942901 20331 raft_consensus.cc:374] T d5c7e23182354fb792aa175f6dcb5f20 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:40.942931 20331 raft_consensus.cc:729] T d5c7e23182354fb792aa175f6dcb5f20 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:40.943097 20331 consensus_queue.cc:260] T d5c7e23182354fb792aa175f6dcb5f20 P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:40.943390 20331 ts_tablet_manager.cc:1398] T d5c7e23182354fb792aa175f6dcb5f20 P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:40.952994 20333 ts_tablet_manager.cc:1398] T d5c7e23182354fb792aa175f6dcb5f20 P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.013s user 0.001s sys 0.000s
I20240503 13:56:40.956532 15472 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 13:56:40.964991 15604 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 13:56:40.967629 13032 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:40.972558 13424 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:56:40.973903 13164 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 13:56:40.981632 19841 consensus_queue.cc:1035] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:40.982537 15868 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 13:56:41.001246 13296 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 13:56:41.001802 15736 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 13:56:41.003221 20232 raft_consensus.cc:482] T d5c7e23182354fb792aa175f6dcb5f20 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:41.003309 20232 raft_consensus.cc:504] T d5c7e23182354fb792aa175f6dcb5f20 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:41.003690 20232 leader_election.cc:288] T d5c7e23182354fb792aa175f6dcb5f20 P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153), 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:41.004880 15493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c7e23182354fb792aa175f6dcb5f20" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546" is_pre_election: true
I20240503 13:56:41.004933 13051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c7e23182354fb792aa175f6dcb5f20" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e" is_pre_election: true
I20240503 13:56:41.005016 15493 raft_consensus.cc:2454] T d5c7e23182354fb792aa175f6dcb5f20 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 0.
I20240503 13:56:41.005044 13051 raft_consensus.cc:2454] T d5c7e23182354fb792aa175f6dcb5f20 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 0.
I20240503 13:56:41.005334 15823 leader_election.cc:302] T d5c7e23182354fb792aa175f6dcb5f20 P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:41.005543 19351 raft_consensus.cc:2781] T d5c7e23182354fb792aa175f6dcb5f20 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:41.005602 19351 raft_consensus.cc:482] T d5c7e23182354fb792aa175f6dcb5f20 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:41.005635 19351 raft_consensus.cc:3037] T d5c7e23182354fb792aa175f6dcb5f20 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:41.006626 19351 raft_consensus.cc:504] T d5c7e23182354fb792aa175f6dcb5f20 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:41.006942 19351 leader_election.cc:288] T d5c7e23182354fb792aa175f6dcb5f20 P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 election: Requested vote from peers 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153), 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:41.007550 13051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c7e23182354fb792aa175f6dcb5f20" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e"
I20240503 13:56:41.007661 13051 raft_consensus.cc:3037] T d5c7e23182354fb792aa175f6dcb5f20 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:41.008641 13051 raft_consensus.cc:2454] T d5c7e23182354fb792aa175f6dcb5f20 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 1.
I20240503 13:56:41.009023 15823 leader_election.cc:302] T d5c7e23182354fb792aa175f6dcb5f20 P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:41.009009 15493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c7e23182354fb792aa175f6dcb5f20" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546"
I20240503 13:56:41.009126 15493 raft_consensus.cc:3037] T d5c7e23182354fb792aa175f6dcb5f20 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:41.010012 15493 raft_consensus.cc:2454] T d5c7e23182354fb792aa175f6dcb5f20 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 1.
I20240503 13:56:41.010197 17953 raft_consensus.cc:2781] T d5c7e23182354fb792aa175f6dcb5f20 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:41.010267 17953 raft_consensus.cc:686] T d5c7e23182354fb792aa175f6dcb5f20 P e47105a0f46e4511b7112e571688a4c7 [term 1 LEADER]: Becoming Leader. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Running, Role: LEADER
I20240503 13:56:41.010385 17953 consensus_queue.cc:237] T d5c7e23182354fb792aa175f6dcb5f20 P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:41.013203 12903 catalog_manager.cc:5484] T d5c7e23182354fb792aa175f6dcb5f20 P e47105a0f46e4511b7112e571688a4c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e47105a0f46e4511b7112e571688a4c7 (127.6.98.72). New cstate: current_term: 1 leader_uuid: "e47105a0f46e4511b7112e571688a4c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:41.022090 20348 ts_tablet_manager.cc:902] T 3f5d536cffe742e69f4b0d3b93caf76c P ba0194098aa84449ae17e8a0d4c43703: Initiating tablet copy from peer 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951)
I20240503 13:56:41.023396 19618 consensus_queue.cc:1035] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:41.028339 20348 tablet_copy_client.cc:308] T 3f5d536cffe742e69f4b0d3b93caf76c P ba0194098aa84449ae17e8a0d4c43703: tablet copy: Beginning tablet copy session from remote peer at address 127.6.98.70:34951
I20240503 13:56:41.029035 15645 tablet_copy_service.cc:140] P 009f849ee7b54c80a3f5264695047e4d: Received BeginTabletCopySession request for tablet 3f5d536cffe742e69f4b0d3b93caf76c from peer ba0194098aa84449ae17e8a0d4c43703 ({username='slave'} at 127.6.98.67:55595)
I20240503 13:56:41.029187 15645 tablet_copy_service.cc:161] P 009f849ee7b54c80a3f5264695047e4d: Beginning new tablet copy session on tablet 3f5d536cffe742e69f4b0d3b93caf76c from peer ba0194098aa84449ae17e8a0d4c43703 at {username='slave'} at 127.6.98.67:55595: session id = ba0194098aa84449ae17e8a0d4c43703-3f5d536cffe742e69f4b0d3b93caf76c
I20240503 13:56:41.030335 15645 tablet_copy_source_session.cc:205] T 3f5d536cffe742e69f4b0d3b93caf76c P 009f849ee7b54c80a3f5264695047e4d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:56:41.031517 20348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f5d536cffe742e69f4b0d3b93caf76c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:41.033592 20348 tablet_copy_client.cc:783] T 3f5d536cffe742e69f4b0d3b93caf76c P ba0194098aa84449ae17e8a0d4c43703: tablet copy: Starting download of 0 data blocks...
I20240503 13:56:41.033740 20348 tablet_copy_client.cc:647] T 3f5d536cffe742e69f4b0d3b93caf76c P ba0194098aa84449ae17e8a0d4c43703: tablet copy: Starting download of 1 WAL segments...
I20240503 13:56:41.035156 20348 tablet_copy_client.cc:523] T 3f5d536cffe742e69f4b0d3b93caf76c P ba0194098aa84449ae17e8a0d4c43703: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:56:41.036320 20348 tablet_bootstrap.cc:492] T 3f5d536cffe742e69f4b0d3b93caf76c P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:41.039626 20348 tablet_bootstrap.cc:492] T 3f5d536cffe742e69f4b0d3b93caf76c P ba0194098aa84449ae17e8a0d4c43703: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:56:41.039870 20348 tablet_bootstrap.cc:492] T 3f5d536cffe742e69f4b0d3b93caf76c P ba0194098aa84449ae17e8a0d4c43703: Bootstrap complete.
I20240503 13:56:41.040045 20348 ts_tablet_manager.cc:1367] T 3f5d536cffe742e69f4b0d3b93caf76c P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240503 13:56:41.040513 20348 raft_consensus.cc:348] T 3f5d536cffe742e69f4b0d3b93caf76c P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:41.040656 20348 raft_consensus.cc:729] T 3f5d536cffe742e69f4b0d3b93caf76c P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:41.040796 20348 consensus_queue.cc:260] T 3f5d536cffe742e69f4b0d3b93caf76c P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:41.041082 20348 ts_tablet_manager.cc:1398] T 3f5d536cffe742e69f4b0d3b93caf76c P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:41.042042 15645 tablet_copy_service.cc:342] P 009f849ee7b54c80a3f5264695047e4d: Request end of tablet copy session ba0194098aa84449ae17e8a0d4c43703-3f5d536cffe742e69f4b0d3b93caf76c received from {username='slave'} at 127.6.98.67:55595
I20240503 13:56:41.042137 15645 tablet_copy_service.cc:433] P 009f849ee7b54c80a3f5264695047e4d: ending tablet copy session ba0194098aa84449ae17e8a0d4c43703-3f5d536cffe742e69f4b0d3b93caf76c on tablet 3f5d536cffe742e69f4b0d3b93caf76c with peer ba0194098aa84449ae17e8a0d4c43703
I20240503 13:56:41.066012 19019 consensus_queue.cc:1035] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:41.069751 19019 consensus_queue.cc:1035] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:41.078079 20379 ts_tablet_manager.cc:902] T c8a53b7095e948f988c9151882bfa1bd P 8ccf5a4ac450414c9d3579bd5800b770: Initiating tablet copy from peer 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951)
I20240503 13:56:41.078584 20379 tablet_copy_client.cc:308] T c8a53b7095e948f988c9151882bfa1bd P 8ccf5a4ac450414c9d3579bd5800b770: tablet copy: Beginning tablet copy session from remote peer at address 127.6.98.70:34951
I20240503 13:56:41.079095 15645 tablet_copy_service.cc:140] P 009f849ee7b54c80a3f5264695047e4d: Received BeginTabletCopySession request for tablet c8a53b7095e948f988c9151882bfa1bd from peer 8ccf5a4ac450414c9d3579bd5800b770 ({username='slave'} at 127.6.98.66:60755)
I20240503 13:56:41.079200 15645 tablet_copy_service.cc:161] P 009f849ee7b54c80a3f5264695047e4d: Beginning new tablet copy session on tablet c8a53b7095e948f988c9151882bfa1bd from peer 8ccf5a4ac450414c9d3579bd5800b770 at {username='slave'} at 127.6.98.66:60755: session id = 8ccf5a4ac450414c9d3579bd5800b770-c8a53b7095e948f988c9151882bfa1bd
I20240503 13:56:41.080205 15645 tablet_copy_source_session.cc:205] T c8a53b7095e948f988c9151882bfa1bd P 009f849ee7b54c80a3f5264695047e4d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:56:41.081152 20379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8a53b7095e948f988c9151882bfa1bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:41.083454 20379 tablet_copy_client.cc:783] T c8a53b7095e948f988c9151882bfa1bd P 8ccf5a4ac450414c9d3579bd5800b770: tablet copy: Starting download of 0 data blocks...
I20240503 13:56:41.083614 20379 tablet_copy_client.cc:647] T c8a53b7095e948f988c9151882bfa1bd P 8ccf5a4ac450414c9d3579bd5800b770: tablet copy: Starting download of 1 WAL segments...
I20240503 13:56:41.084926 20379 tablet_copy_client.cc:523] T c8a53b7095e948f988c9151882bfa1bd P 8ccf5a4ac450414c9d3579bd5800b770: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:56:41.086035 20379 tablet_bootstrap.cc:492] T c8a53b7095e948f988c9151882bfa1bd P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:41.089491 20379 tablet_bootstrap.cc:492] T c8a53b7095e948f988c9151882bfa1bd P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:56:41.089699 20379 tablet_bootstrap.cc:492] T c8a53b7095e948f988c9151882bfa1bd P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap complete.
I20240503 13:56:41.089843 20379 ts_tablet_manager.cc:1367] T c8a53b7095e948f988c9151882bfa1bd P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 13:56:41.090258 20379 raft_consensus.cc:348] T c8a53b7095e948f988c9151882bfa1bd P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:41.090370 20379 raft_consensus.cc:729] T c8a53b7095e948f988c9151882bfa1bd P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:41.090492 20379 consensus_queue.cc:260] T c8a53b7095e948f988c9151882bfa1bd P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:41.090787 20379 ts_tablet_manager.cc:1398] T c8a53b7095e948f988c9151882bfa1bd P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:41.092020 15645 tablet_copy_service.cc:342] P 009f849ee7b54c80a3f5264695047e4d: Request end of tablet copy session 8ccf5a4ac450414c9d3579bd5800b770-c8a53b7095e948f988c9151882bfa1bd received from {username='slave'} at 127.6.98.66:60755
I20240503 13:56:41.092108 15645 tablet_copy_service.cc:433] P 009f849ee7b54c80a3f5264695047e4d: ending tablet copy session 8ccf5a4ac450414c9d3579bd5800b770-c8a53b7095e948f988c9151882bfa1bd on tablet c8a53b7095e948f988c9151882bfa1bd with peer 8ccf5a4ac450414c9d3579bd5800b770
I20240503 13:56:41.111371 20410 ts_tablet_manager.cc:902] T 258d1ec1f0694c15a4d41bd064f35b4e P 61b887f0d40d497badeca5604c1f0546: Initiating tablet copy from peer ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:41.112041 20410 tablet_copy_client.cc:308] T 258d1ec1f0694c15a4d41bd064f35b4e P 61b887f0d40d497badeca5604c1f0546: tablet copy: Beginning tablet copy session from remote peer at address 127.6.98.67:42279
I20240503 13:56:41.113505 13336 tablet_copy_service.cc:140] P ba0194098aa84449ae17e8a0d4c43703: Received BeginTabletCopySession request for tablet 258d1ec1f0694c15a4d41bd064f35b4e from peer 61b887f0d40d497badeca5604c1f0546 ({username='slave'} at 127.6.98.69:36581)
I20240503 13:56:41.113613 13336 tablet_copy_service.cc:161] P ba0194098aa84449ae17e8a0d4c43703: Beginning new tablet copy session on tablet 258d1ec1f0694c15a4d41bd064f35b4e from peer 61b887f0d40d497badeca5604c1f0546 at {username='slave'} at 127.6.98.69:36581: session id = 61b887f0d40d497badeca5604c1f0546-258d1ec1f0694c15a4d41bd064f35b4e
I20240503 13:56:41.114655 13336 tablet_copy_source_session.cc:205] T 258d1ec1f0694c15a4d41bd064f35b4e P ba0194098aa84449ae17e8a0d4c43703: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:56:41.117826 12903 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:56:41.119323 20410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 258d1ec1f0694c15a4d41bd064f35b4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:41.126258 15493 raft_consensus.cc:1261] T d5c7e23182354fb792aa175f6dcb5f20 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Refusing update from remote peer e47105a0f46e4511b7112e571688a4c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:41.127064 20232 consensus_queue.cc:1035] T d5c7e23182354fb792aa175f6dcb5f20 P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:41.128945 13051 raft_consensus.cc:1261] T d5c7e23182354fb792aa175f6dcb5f20 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Refusing update from remote peer e47105a0f46e4511b7112e571688a4c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:41.129650 20232 consensus_queue.cc:1035] T d5c7e23182354fb792aa175f6dcb5f20 P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:41.131278 12906 catalog_manager.cc:5576] TS 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153) does not have the latest schema for tablet d5c7e23182354fb792aa175f6dcb5f20 (table rebalancer_dynamic_table [id=2a01aba75336445ca17e5ef317d14b36]). Expected version 1 got 0
I20240503 13:56:41.132663 19019 consensus_queue.cc:1035] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:41.138231 12904 catalog_manager.cc:5576] TS 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695) does not have the latest schema for tablet d5c7e23182354fb792aa175f6dcb5f20 (table rebalancer_dynamic_table [id=2a01aba75336445ca17e5ef317d14b36]). Expected version 1 got 0
I20240503 13:56:41.139075 19998 consensus_queue.cc:1035] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:41.143762 20424 mvcc.cc:205] Tried to move back new op lower bound from 7023593886202912768 to 7023593885739757568. Current Snapshot: MvccSnapshot[applied={T|T < 7023593886202912768}]
I20240503 13:56:41.144904 20429 tablet.cc:1722] T d5c7e23182354fb792aa175f6dcb5f20 P 60ba3ffe3b7746469834cd91c84be14e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:41.145260 20429 tablet.cc:1620] T d5c7e23182354fb792aa175f6dcb5f20 P 60ba3ffe3b7746469834cd91c84be14e: MemRowSet was empty: no flush needed.
I20240503 13:56:41.146277 20123 tablet.cc:1722] T d5c7e23182354fb792aa175f6dcb5f20 P 61b887f0d40d497badeca5604c1f0546: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:41.146560 20123 tablet.cc:1620] T d5c7e23182354fb792aa175f6dcb5f20 P 61b887f0d40d497badeca5604c1f0546: MemRowSet was empty: no flush needed.
I20240503 13:56:41.147648 12910 catalog_manager.cc:5576] TS 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153) does not have the latest schema for tablet d5c7e23182354fb792aa175f6dcb5f20 (table rebalancer_dynamic_table [id=2a01aba75336445ca17e5ef317d14b36]). Expected version 1 got 0
I20240503 13:56:41.148952 12909 catalog_manager.cc:5576] TS 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695) does not have the latest schema for tablet d5c7e23182354fb792aa175f6dcb5f20 (table rebalancer_dynamic_table [id=2a01aba75336445ca17e5ef317d14b36]). Expected version 1 got 0
I20240503 13:56:41.149996 20410 tablet_copy_client.cc:783] T 258d1ec1f0694c15a4d41bd064f35b4e P 61b887f0d40d497badeca5604c1f0546: tablet copy: Starting download of 0 data blocks...
I20240503 13:56:41.150223 20410 tablet_copy_client.cc:647] T 258d1ec1f0694c15a4d41bd064f35b4e P 61b887f0d40d497badeca5604c1f0546: tablet copy: Starting download of 1 WAL segments...
I20240503 13:56:41.150907 20427 tablet.cc:1722] T d5c7e23182354fb792aa175f6dcb5f20 P e47105a0f46e4511b7112e571688a4c7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:41.151293 20427 tablet.cc:1620] T d5c7e23182354fb792aa175f6dcb5f20 P e47105a0f46e4511b7112e571688a4c7: MemRowSet was empty: no flush needed.
I20240503 13:56:41.153759 12912 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=2a01aba75336445ca17e5ef317d14b36] alter complete (version 1)
I20240503 13:56:41.154008 20410 tablet_copy_client.cc:523] T 258d1ec1f0694c15a4d41bd064f35b4e P 61b887f0d40d497badeca5604c1f0546: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:56:41.156651 20410 tablet_bootstrap.cc:492] T 258d1ec1f0694c15a4d41bd064f35b4e P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:41.158244 12912 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_extra_table_32"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:56:41.167068 15868 tablet_service.cc:1467] Processing CreateTablet for tablet 3be039d9c2294eb5bec575fc0505280c (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=560f038428be4ca087281f51df1b0a85]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:41.167369 15868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3be039d9c2294eb5bec575fc0505280c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:41.167426 15867 tablet_service.cc:1467] Processing CreateTablet for tablet 76f1450a07f34baaa8ecab16340e0ee0 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=560f038428be4ca087281f51df1b0a85]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:41.167667 15867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76f1450a07f34baaa8ecab16340e0ee0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:41.167956 13296 tablet_service.cc:1467] Processing CreateTablet for tablet 37d81c890a1d411bb496dfe843bbd7e7 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=560f038428be4ca087281f51df1b0a85]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:41.168215 13296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37d81c890a1d411bb496dfe843bbd7e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:41.168795 13032 tablet_service.cc:1467] Processing CreateTablet for tablet 76f1450a07f34baaa8ecab16340e0ee0 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=560f038428be4ca087281f51df1b0a85]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:41.169049 13032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76f1450a07f34baaa8ecab16340e0ee0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:41.169591 20135 tablet_bootstrap.cc:492] T 3be039d9c2294eb5bec575fc0505280c P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:41.170527 20135 tablet_bootstrap.cc:654] T 3be039d9c2294eb5bec575fc0505280c P e47105a0f46e4511b7112e571688a4c7: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:41.171118 15472 tablet_service.cc:1467] Processing CreateTablet for tablet 3be039d9c2294eb5bec575fc0505280c (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=560f038428be4ca087281f51df1b0a85]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:41.171363 15472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3be039d9c2294eb5bec575fc0505280c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:41.171690 15736 tablet_service.cc:1467] Processing CreateTablet for tablet 37d81c890a1d411bb496dfe843bbd7e7 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=560f038428be4ca087281f51df1b0a85]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:41.171952 15736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37d81c890a1d411bb496dfe843bbd7e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:41.172277 20431 tablet_bootstrap.cc:492] T 37d81c890a1d411bb496dfe843bbd7e7 P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:41.172724 15738 tablet_service.cc:1467] Processing CreateTablet for tablet 76f1450a07f34baaa8ecab16340e0ee0 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=560f038428be4ca087281f51df1b0a85]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:41.172951 15738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76f1450a07f34baaa8ecab16340e0ee0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:41.173252 20333 tablet_bootstrap.cc:492] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:41.173446 20331 tablet_bootstrap.cc:492] T 76f1450a07f34baaa8ecab16340e0ee0 P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:41.174017 20431 tablet_bootstrap.cc:654] T 37d81c890a1d411bb496dfe843bbd7e7 P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:41.174175 20333 tablet_bootstrap.cc:654] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:41.174496 20331 tablet_bootstrap.cc:654] T 76f1450a07f34baaa8ecab16340e0ee0 P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:41.176170 20431 tablet_bootstrap.cc:492] T 37d81c890a1d411bb496dfe843bbd7e7 P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:41.176682 20333 tablet_bootstrap.cc:492] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546: No bootstrap required, opened a new log
I20240503 13:56:41.176756 20333 ts_tablet_manager.cc:1367] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:56:41.176872 20135 tablet_bootstrap.cc:492] T 3be039d9c2294eb5bec575fc0505280c P e47105a0f46e4511b7112e571688a4c7: No bootstrap required, opened a new log
I20240503 13:56:41.176954 20135 ts_tablet_manager.cc:1367] T 3be039d9c2294eb5bec575fc0505280c P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20240503 13:56:41.177173 20333 raft_consensus.cc:348] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:41.177279 20333 raft_consensus.cc:374] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:41.177307 20333 raft_consensus.cc:729] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:41.177376 20135 raft_consensus.cc:348] T 3be039d9c2294eb5bec575fc0505280c P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:41.177414 20333 consensus_queue.cc:260] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:41.177471 20135 raft_consensus.cc:374] T 3be039d9c2294eb5bec575fc0505280c P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:41.177500 20135 raft_consensus.cc:729] T 3be039d9c2294eb5bec575fc0505280c P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:41.177677 20333 ts_tablet_manager.cc:1398] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:41.177625 20135 consensus_queue.cc:260] T 3be039d9c2294eb5bec575fc0505280c P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:41.177851 20431 ts_tablet_manager.cc:1367] T 37d81c890a1d411bb496dfe843bbd7e7 P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 13:56:41.177907 20135 ts_tablet_manager.cc:1398] T 3be039d9c2294eb5bec575fc0505280c P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:41.178347 20431 raft_consensus.cc:348] T 37d81c890a1d411bb496dfe843bbd7e7 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:41.178840 15604 tablet_service.cc:1467] Processing CreateTablet for tablet 37d81c890a1d411bb496dfe843bbd7e7 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=560f038428be4ca087281f51df1b0a85]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:41.178449 20431 raft_consensus.cc:374] T 37d81c890a1d411bb496dfe843bbd7e7 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:41.179090 20431 raft_consensus.cc:729] T 37d81c890a1d411bb496dfe843bbd7e7 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:41.179118 15604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37d81c890a1d411bb496dfe843bbd7e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:41.179486 20431 consensus_queue.cc:260] T 37d81c890a1d411bb496dfe843bbd7e7 P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:41.179790 20431 ts_tablet_manager.cc:1398] T 37d81c890a1d411bb496dfe843bbd7e7 P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:41.180367 20331 tablet_bootstrap.cc:492] T 76f1450a07f34baaa8ecab16340e0ee0 P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:41.180445 20331 ts_tablet_manager.cc:1367] T 76f1450a07f34baaa8ecab16340e0ee0 P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 13:56:41.180888 20331 raft_consensus.cc:348] T 76f1450a07f34baaa8ecab16340e0ee0 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:41.180981 20331 raft_consensus.cc:374] T 76f1450a07f34baaa8ecab16340e0ee0 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:41.181011 20331 raft_consensus.cc:729] T 76f1450a07f34baaa8ecab16340e0ee0 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:41.181125 20331 consensus_queue.cc:260] T 76f1450a07f34baaa8ecab16340e0ee0 P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:41.181392 20331 ts_tablet_manager.cc:1398] T 76f1450a07f34baaa8ecab16340e0ee0 P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:41.181599 15605 tablet_service.cc:1467] Processing CreateTablet for tablet 3be039d9c2294eb5bec575fc0505280c (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=560f038428be4ca087281f51df1b0a85]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:41.181823 15605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3be039d9c2294eb5bec575fc0505280c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:41.193990 13424 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:56:41.196743 20410 tablet_bootstrap.cc:492] T 258d1ec1f0694c15a4d41bd064f35b4e P 61b887f0d40d497badeca5604c1f0546: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:56:41.196993 20410 tablet_bootstrap.cc:492] T 258d1ec1f0694c15a4d41bd064f35b4e P 61b887f0d40d497badeca5604c1f0546: Bootstrap complete.
I20240503 13:56:41.197175 20410 ts_tablet_manager.cc:1367] T 258d1ec1f0694c15a4d41bd064f35b4e P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.041s user 0.021s sys 0.003s
I20240503 13:56:41.197652 20410 raft_consensus.cc:348] T 258d1ec1f0694c15a4d41bd064f35b4e P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:41.197768 20410 raft_consensus.cc:729] T 258d1ec1f0694c15a4d41bd064f35b4e P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:41.197888 20410 consensus_queue.cc:260] T 258d1ec1f0694c15a4d41bd064f35b4e P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:41.198164 20410 ts_tablet_manager.cc:1398] T 258d1ec1f0694c15a4d41bd064f35b4e P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:41.198416 15472 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 13:56:41.198881 13336 tablet_copy_service.cc:342] P ba0194098aa84449ae17e8a0d4c43703: Request end of tablet copy session 61b887f0d40d497badeca5604c1f0546-258d1ec1f0694c15a4d41bd064f35b4e received from {username='slave'} at 127.6.98.69:36581
I20240503 13:56:41.198962 13336 tablet_copy_service.cc:433] P ba0194098aa84449ae17e8a0d4c43703: ending tablet copy session 61b887f0d40d497badeca5604c1f0546-258d1ec1f0694c15a4d41bd064f35b4e on tablet 258d1ec1f0694c15a4d41bd064f35b4e with peer 61b887f0d40d497badeca5604c1f0546
I20240503 13:56:41.205757 18914 raft_consensus.cc:482] T 37d81c890a1d411bb496dfe843bbd7e7 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:41.205881 18914 raft_consensus.cc:504] T 37d81c890a1d411bb496dfe843bbd7e7 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:41.206331 18914 leader_election.cc:288] T 37d81c890a1d411bb496dfe843bbd7e7 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885), 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951)
I20240503 13:56:41.207065 15758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37d81c890a1d411bb496dfe843bbd7e7" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2" is_pre_election: true
W20240503 13:56:41.207604 13252 leader_election.cc:341] T 37d81c890a1d411bb496dfe843bbd7e7 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885): Illegal state: must be running to vote when last-logged opid is not known
I20240503 13:56:41.208150 15624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37d81c890a1d411bb496dfe843bbd7e7" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d" is_pre_election: true
W20240503 13:56:41.208856 13249 leader_election.cc:341] T 37d81c890a1d411bb496dfe843bbd7e7 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951): Illegal state: must be running to vote when last-logged opid is not known
I20240503 13:56:41.208927 13249 leader_election.cc:302] T 37d81c890a1d411bb496dfe843bbd7e7 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba0194098aa84449ae17e8a0d4c43703; no voters: 009f849ee7b54c80a3f5264695047e4d, 76c386bef61a4bd5a907266a5118ffe2
I20240503 13:56:41.209071 18914 raft_consensus.cc:2726] T 37d81c890a1d411bb496dfe843bbd7e7 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 13:56:41.215250 15605 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 13:56:41.222560 20346 raft_consensus.cc:482] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:41.222674 20346 raft_consensus.cc:504] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:41.223079 20346 leader_election.cc:288] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951)
I20240503 13:56:41.223443 15888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3be039d9c2294eb5bec575fc0505280c" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7" is_pre_election: true
I20240503 13:56:41.223546 15888 raft_consensus.cc:2454] T 3be039d9c2294eb5bec575fc0505280c P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 0.
I20240503 13:56:41.223788 15426 leader_election.cc:302] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61b887f0d40d497badeca5604c1f0546, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:41.223950 20346 raft_consensus.cc:2781] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:41.223984 20346 raft_consensus.cc:482] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:41.224006 20346 raft_consensus.cc:3037] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:41.224452 15623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3be039d9c2294eb5bec575fc0505280c" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d" is_pre_election: true
I20240503 13:56:41.224902 20346 raft_consensus.cc:504] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:41.225190 20346 leader_election.cc:288] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 election: Requested vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951)
W20240503 13:56:41.226231 15426 leader_election.cc:341] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951): Illegal state: must be running to vote when last-logged opid is not known
I20240503 13:56:41.226393 15888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3be039d9c2294eb5bec575fc0505280c" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7"
I20240503 13:56:41.226513 15888 raft_consensus.cc:3037] T 3be039d9c2294eb5bec575fc0505280c P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:41.226588 15623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3be039d9c2294eb5bec575fc0505280c" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d"
I20240503 13:56:41.227806 15888 raft_consensus.cc:2454] T 3be039d9c2294eb5bec575fc0505280c P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 1.
I20240503 13:56:41.229817 15426 leader_election.cc:302] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61b887f0d40d497badeca5604c1f0546, e47105a0f46e4511b7112e571688a4c7; no voters:
W20240503 13:56:41.230012 15426 leader_election.cc:341] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951): Illegal state: must be running to vote when last-logged opid is not known
I20240503 13:56:41.230218 20346 raft_consensus.cc:2781] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:41.230299 20346 raft_consensus.cc:686] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 [term 1 LEADER]: Becoming Leader. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Running, Role: LEADER
I20240503 13:56:41.230428 20346 consensus_queue.cc:237] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:41.233654 13032 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:41.236214 12912 catalog_manager.cc:5484] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61b887f0d40d497badeca5604c1f0546 (127.6.98.69). New cstate: current_term: 1 leader_uuid: "61b887f0d40d497badeca5604c1f0546" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:41.237713 20135 tablet_bootstrap.cc:492] T 76f1450a07f34baaa8ecab16340e0ee0 P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:41.237915 15736 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 13:56:41.238857 20135 tablet_bootstrap.cc:654] T 76f1450a07f34baaa8ecab16340e0ee0 P e47105a0f46e4511b7112e571688a4c7: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:41.250387 13296 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 13:56:41.251176 13164 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 13:56:41.255681 15867 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:56:41.256934 20446 tablet_bootstrap.cc:492] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:41.258123 20448 tablet_bootstrap.cc:492] T 37d81c890a1d411bb496dfe843bbd7e7 P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:41.259202 20448 tablet_bootstrap.cc:654] T 37d81c890a1d411bb496dfe843bbd7e7 P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:41.259722 12912 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:56:41.261164 20446 tablet_bootstrap.cc:654] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:41.265745 20452 ts_tablet_manager.cc:902] T eb92a057165f4da7b96d910d847ff761 P 76c386bef61a4bd5a907266a5118ffe2: Initiating tablet copy from peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:41.266306 20452 tablet_copy_client.cc:308] T eb92a057165f4da7b96d910d847ff761 P 76c386bef61a4bd5a907266a5118ffe2: tablet copy: Beginning tablet copy session from remote peer at address 127.6.98.72:44753
I20240503 13:56:41.267356 20448 tablet_bootstrap.cc:492] T 37d81c890a1d411bb496dfe843bbd7e7 P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:41.267550 20448 ts_tablet_manager.cc:1367] T 37d81c890a1d411bb496dfe843bbd7e7 P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 13:56:41.267894 15909 tablet_copy_service.cc:140] P e47105a0f46e4511b7112e571688a4c7: Received BeginTabletCopySession request for tablet eb92a057165f4da7b96d910d847ff761 from peer 76c386bef61a4bd5a907266a5118ffe2 ({username='slave'} at 127.6.98.71:60433)
I20240503 13:56:41.267997 15909 tablet_copy_service.cc:161] P e47105a0f46e4511b7112e571688a4c7: Beginning new tablet copy session on tablet eb92a057165f4da7b96d910d847ff761 from peer 76c386bef61a4bd5a907266a5118ffe2 at {username='slave'} at 127.6.98.71:60433: session id = 76c386bef61a4bd5a907266a5118ffe2-eb92a057165f4da7b96d910d847ff761
I20240503 13:56:41.268061 20448 raft_consensus.cc:348] T 37d81c890a1d411bb496dfe843bbd7e7 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:41.268170 20448 raft_consensus.cc:374] T 37d81c890a1d411bb496dfe843bbd7e7 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:41.268196 20448 raft_consensus.cc:729] T 37d81c890a1d411bb496dfe843bbd7e7 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:41.268309 20448 consensus_queue.cc:260] T 37d81c890a1d411bb496dfe843bbd7e7 P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:41.268637 20448 ts_tablet_manager.cc:1398] T 37d81c890a1d411bb496dfe843bbd7e7 P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:41.268952 15909 tablet_copy_source_session.cc:205] T eb92a057165f4da7b96d910d847ff761 P e47105a0f46e4511b7112e571688a4c7: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:56:41.271129 20429 tablet.cc:1722] T d5c7e23182354fb792aa175f6dcb5f20 P 60ba3ffe3b7746469834cd91c84be14e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:41.271447 20429 tablet.cc:1620] T d5c7e23182354fb792aa175f6dcb5f20 P 60ba3ffe3b7746469834cd91c84be14e: MemRowSet was empty: no flush needed.
I20240503 13:56:41.273263 20449 ts_tablet_manager.cc:902] T b1395d66d8e04ecdafd7a30c50aa1c9a P 009f849ee7b54c80a3f5264695047e4d: Initiating tablet copy from peer 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:41.273624 20452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb92a057165f4da7b96d910d847ff761. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:41.274602 20449 tablet_copy_client.cc:308] T b1395d66d8e04ecdafd7a30c50aa1c9a P 009f849ee7b54c80a3f5264695047e4d: tablet copy: Beginning tablet copy session from remote peer at address 127.6.98.68:36109
I20240503 13:56:41.274741 20448 tablet_bootstrap.cc:492] T 3be039d9c2294eb5bec575fc0505280c P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:41.275106 13467 tablet_copy_service.cc:140] P 7690ee79577c45e3a2c8f8b552c0bff8: Received BeginTabletCopySession request for tablet b1395d66d8e04ecdafd7a30c50aa1c9a from peer 009f849ee7b54c80a3f5264695047e4d ({username='slave'} at 127.6.98.70:40885)
I20240503 13:56:41.275228 13467 tablet_copy_service.cc:161] P 7690ee79577c45e3a2c8f8b552c0bff8: Beginning new tablet copy session on tablet b1395d66d8e04ecdafd7a30c50aa1c9a from peer 009f849ee7b54c80a3f5264695047e4d at {username='slave'} at 127.6.98.70:40885: session id = 009f849ee7b54c80a3f5264695047e4d-b1395d66d8e04ecdafd7a30c50aa1c9a
I20240503 13:56:41.275921 20448 tablet_bootstrap.cc:654] T 3be039d9c2294eb5bec575fc0505280c P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:41.276171 13467 tablet_copy_source_session.cc:205] T b1395d66d8e04ecdafd7a30c50aa1c9a P 7690ee79577c45e3a2c8f8b552c0bff8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:56:41.277494 12911 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=2a01aba75336445ca17e5ef317d14b36] alter complete (version 2)
I20240503 13:56:41.278085 20449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1395d66d8e04ecdafd7a30c50aa1c9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:41.278868 20123 tablet.cc:1722] T d5c7e23182354fb792aa175f6dcb5f20 P 61b887f0d40d497badeca5604c1f0546: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:41.279168 20123 tablet.cc:1620] T d5c7e23182354fb792aa175f6dcb5f20 P 61b887f0d40d497badeca5604c1f0546: MemRowSet was empty: no flush needed.
I20240503 13:56:41.286017 20452 tablet_copy_client.cc:783] T eb92a057165f4da7b96d910d847ff761 P 76c386bef61a4bd5a907266a5118ffe2: tablet copy: Starting download of 0 data blocks...
I20240503 13:56:41.286242 20452 tablet_copy_client.cc:647] T eb92a057165f4da7b96d910d847ff761 P 76c386bef61a4bd5a907266a5118ffe2: tablet copy: Starting download of 1 WAL segments...
I20240503 13:56:41.286525 20449 tablet_copy_client.cc:783] T b1395d66d8e04ecdafd7a30c50aa1c9a P 009f849ee7b54c80a3f5264695047e4d: tablet copy: Starting download of 0 data blocks...
I20240503 13:56:41.286741 20449 tablet_copy_client.cc:647] T b1395d66d8e04ecdafd7a30c50aa1c9a P 009f849ee7b54c80a3f5264695047e4d: tablet copy: Starting download of 1 WAL segments...
I20240503 13:56:41.287289 20452 tablet_copy_client.cc:523] T eb92a057165f4da7b96d910d847ff761 P 76c386bef61a4bd5a907266a5118ffe2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:56:41.288652 20452 tablet_bootstrap.cc:492] T eb92a057165f4da7b96d910d847ff761 P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:41.289754 20449 tablet_copy_client.cc:523] T b1395d66d8e04ecdafd7a30c50aa1c9a P 009f849ee7b54c80a3f5264695047e4d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:56:41.290339 20448 tablet_bootstrap.cc:492] T 3be039d9c2294eb5bec575fc0505280c P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:41.290474 20448 ts_tablet_manager.cc:1367] T 3be039d9c2294eb5bec575fc0505280c P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240503 13:56:41.291232 20449 tablet_bootstrap.cc:492] T b1395d66d8e04ecdafd7a30c50aa1c9a P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:41.292062 20448 raft_consensus.cc:348] T 3be039d9c2294eb5bec575fc0505280c P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:41.292230 20448 raft_consensus.cc:374] T 3be039d9c2294eb5bec575fc0505280c P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:41.292294 20448 raft_consensus.cc:729] T 3be039d9c2294eb5bec575fc0505280c P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:41.292922 20448 consensus_queue.cc:260] T 3be039d9c2294eb5bec575fc0505280c P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:41.293293 20448 ts_tablet_manager.cc:1398] T 3be039d9c2294eb5bec575fc0505280c P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:56:41.299734 20452 tablet_bootstrap.cc:492] T eb92a057165f4da7b96d910d847ff761 P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:56:41.299974 20452 tablet_bootstrap.cc:492] T eb92a057165f4da7b96d910d847ff761 P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap complete.
I20240503 13:56:41.300118 20452 ts_tablet_manager.cc:1367] T eb92a057165f4da7b96d910d847ff761 P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.001s
I20240503 13:56:41.300557 20452 raft_consensus.cc:348] T eb92a057165f4da7b96d910d847ff761 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:41.300662 20452 raft_consensus.cc:729] T eb92a057165f4da7b96d910d847ff761 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:41.300772 20452 consensus_queue.cc:260] T eb92a057165f4da7b96d910d847ff761 P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:41.301023 20452 ts_tablet_manager.cc:1398] T eb92a057165f4da7b96d910d847ff761 P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:41.302310 15909 tablet_copy_service.cc:342] P e47105a0f46e4511b7112e571688a4c7: Request end of tablet copy session 76c386bef61a4bd5a907266a5118ffe2-eb92a057165f4da7b96d910d847ff761 received from {username='slave'} at 127.6.98.71:60433
I20240503 13:56:41.302397 15909 tablet_copy_service.cc:433] P e47105a0f46e4511b7112e571688a4c7: ending tablet copy session 76c386bef61a4bd5a907266a5118ffe2-eb92a057165f4da7b96d910d847ff761 on tablet eb92a057165f4da7b96d910d847ff761 with peer 76c386bef61a4bd5a907266a5118ffe2
I20240503 13:56:41.311269 20427 tablet.cc:1722] T d5c7e23182354fb792aa175f6dcb5f20 P e47105a0f46e4511b7112e571688a4c7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:41.311637 20427 tablet.cc:1620] T d5c7e23182354fb792aa175f6dcb5f20 P e47105a0f46e4511b7112e571688a4c7: MemRowSet was empty: no flush needed.
I20240503 13:56:41.312592 20135 tablet_bootstrap.cc:492] T 76f1450a07f34baaa8ecab16340e0ee0 P e47105a0f46e4511b7112e571688a4c7: No bootstrap required, opened a new log
I20240503 13:56:41.312700 20135 ts_tablet_manager.cc:1367] T 76f1450a07f34baaa8ecab16340e0ee0 P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.075s user 0.002s sys 0.000s
I20240503 13:56:41.313192 20135 raft_consensus.cc:348] T 76f1450a07f34baaa8ecab16340e0ee0 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:41.313318 20135 raft_consensus.cc:374] T 76f1450a07f34baaa8ecab16340e0ee0 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:41.313357 20135 raft_consensus.cc:729] T 76f1450a07f34baaa8ecab16340e0ee0 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:41.313481 20135 consensus_queue.cc:260] T 76f1450a07f34baaa8ecab16340e0ee0 P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:41.313817 20135 ts_tablet_manager.cc:1398] T 76f1450a07f34baaa8ecab16340e0ee0 P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:41.316327 20446 tablet_bootstrap.cc:492] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:41.316419 20446 ts_tablet_manager.cc:1367] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.060s user 0.001s sys 0.001s
I20240503 13:56:41.316874 20446 raft_consensus.cc:348] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:41.316978 20446 raft_consensus.cc:374] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:41.317004 20446 raft_consensus.cc:729] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:41.317117 20446 consensus_queue.cc:260] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:41.317373 20446 ts_tablet_manager.cc:1398] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:41.317536 20446 tablet_bootstrap.cc:492] T 37d81c890a1d411bb496dfe843bbd7e7 P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:41.318410 20446 tablet_bootstrap.cc:654] T 37d81c890a1d411bb496dfe843bbd7e7 P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:41.329694 20446 tablet_bootstrap.cc:492] T 37d81c890a1d411bb496dfe843bbd7e7 P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:41.329821 20446 ts_tablet_manager.cc:1367] T 37d81c890a1d411bb496dfe843bbd7e7 P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.001s
I20240503 13:56:41.330344 20446 raft_consensus.cc:348] T 37d81c890a1d411bb496dfe843bbd7e7 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:41.330471 20446 raft_consensus.cc:374] T 37d81c890a1d411bb496dfe843bbd7e7 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:41.330519 20446 raft_consensus.cc:729] T 37d81c890a1d411bb496dfe843bbd7e7 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:41.330654 20446 consensus_queue.cc:260] T 37d81c890a1d411bb496dfe843bbd7e7 P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:41.330978 20446 ts_tablet_manager.cc:1398] T 37d81c890a1d411bb496dfe843bbd7e7 P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 13:56:41.337973 20449 tablet_bootstrap.cc:492] T b1395d66d8e04ecdafd7a30c50aa1c9a P 009f849ee7b54c80a3f5264695047e4d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:56:41.338272 20449 tablet_bootstrap.cc:492] T b1395d66d8e04ecdafd7a30c50aa1c9a P 009f849ee7b54c80a3f5264695047e4d: Bootstrap complete.
I20240503 13:56:41.338462 20449 ts_tablet_manager.cc:1367] T b1395d66d8e04ecdafd7a30c50aa1c9a P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.047s user 0.019s sys 0.003s
I20240503 13:56:41.338909 20449 raft_consensus.cc:348] T b1395d66d8e04ecdafd7a30c50aa1c9a P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:41.339104 20449 raft_consensus.cc:729] T b1395d66d8e04ecdafd7a30c50aa1c9a P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:41.339288 20449 consensus_queue.cc:260] T b1395d66d8e04ecdafd7a30c50aa1c9a P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:41.339691 20449 ts_tablet_manager.cc:1398] T b1395d66d8e04ecdafd7a30c50aa1c9a P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 13:56:41.340130 13467 tablet_copy_service.cc:342] P 7690ee79577c45e3a2c8f8b552c0bff8: Request end of tablet copy session 009f849ee7b54c80a3f5264695047e4d-b1395d66d8e04ecdafd7a30c50aa1c9a received from {username='slave'} at 127.6.98.70:40885
I20240503 13:56:41.340206 13467 tablet_copy_service.cc:433] P 7690ee79577c45e3a2c8f8b552c0bff8: ending tablet copy session 009f849ee7b54c80a3f5264695047e4d-b1395d66d8e04ecdafd7a30c50aa1c9a on tablet b1395d66d8e04ecdafd7a30c50aa1c9a with peer 009f849ee7b54c80a3f5264695047e4d
I20240503 13:56:41.379228 12906 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:56:41.390640 20427 tablet.cc:1722] T d5c7e23182354fb792aa175f6dcb5f20 P e47105a0f46e4511b7112e571688a4c7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:41.391381 20429 tablet.cc:1722] T d5c7e23182354fb792aa175f6dcb5f20 P 60ba3ffe3b7746469834cd91c84be14e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:41.393144 20123 tablet.cc:1722] T d5c7e23182354fb792aa175f6dcb5f20 P 61b887f0d40d497badeca5604c1f0546: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:41.396972 12907 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table_new_name [id=2a01aba75336445ca17e5ef317d14b36] alter complete (version 3)
I20240503 13:56:41.412139 19841 raft_consensus.cc:482] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:41.412277 19841 raft_consensus.cc:504] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:41.412720 19841 leader_election.cc:288] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:41.413017 15888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76f1450a07f34baaa8ecab16340e0ee0" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7" is_pre_election: true
I20240503 13:56:41.413132 15888 raft_consensus.cc:2454] T 76f1450a07f34baaa8ecab16340e0ee0 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 0.
I20240503 13:56:41.413424 13051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76f1450a07f34baaa8ecab16340e0ee0" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e" is_pre_election: true
I20240503 13:56:41.413547 13051 raft_consensus.cc:2454] T 76f1450a07f34baaa8ecab16340e0ee0 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 0.
I20240503 13:56:41.414220 15692 leader_election.cc:302] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 76c386bef61a4bd5a907266a5118ffe2; no voters:
I20240503 13:56:41.414781 19841 raft_consensus.cc:2781] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:41.414836 19841 raft_consensus.cc:482] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:41.414881 19841 raft_consensus.cc:3037] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:41.416036 19841 raft_consensus.cc:504] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:41.416388 19841 leader_election.cc:288] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 election: Requested vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:41.417793 13051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76f1450a07f34baaa8ecab16340e0ee0" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e"
I20240503 13:56:41.417836 15888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76f1450a07f34baaa8ecab16340e0ee0" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7"
I20240503 13:56:41.417913 13051 raft_consensus.cc:3037] T 76f1450a07f34baaa8ecab16340e0ee0 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:41.417937 15888 raft_consensus.cc:3037] T 76f1450a07f34baaa8ecab16340e0ee0 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:41.418838 13051 raft_consensus.cc:2454] T 76f1450a07f34baaa8ecab16340e0ee0 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 1.
I20240503 13:56:41.418838 15888 raft_consensus.cc:2454] T 76f1450a07f34baaa8ecab16340e0ee0 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 1.
I20240503 13:56:41.419121 15692 leader_election.cc:302] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 76c386bef61a4bd5a907266a5118ffe2; no voters:
I20240503 13:56:41.419796 19841 raft_consensus.cc:2781] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:41.419873 19841 raft_consensus.cc:686] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 LEADER]: Becoming Leader. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Running, Role: LEADER
I20240503 13:56:41.419998 19841 consensus_queue.cc:237] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:41.422319 13314 raft_consensus.cc:1203] T 3f5d536cffe742e69f4b0d3b93caf76c P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:56:41.423664 12911 catalog_manager.cc:5484] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71). New cstate: current_term: 1 leader_uuid: "76c386bef61a4bd5a907266a5118ffe2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } }
I20240503 13:56:41.492362 15492 raft_consensus.cc:1203] T 258d1ec1f0694c15a4d41bd064f35b4e P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:56:41.495524 19586 raft_consensus.cc:482] T 37d81c890a1d411bb496dfe843bbd7e7 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:41.495640 19586 raft_consensus.cc:504] T 37d81c890a1d411bb496dfe843bbd7e7 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:41.496049 19586 leader_election.cc:288] T 37d81c890a1d411bb496dfe843bbd7e7 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885), ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:41.496445 15758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37d81c890a1d411bb496dfe843bbd7e7" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2" is_pre_election: true
I20240503 13:56:41.496465 13314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37d81c890a1d411bb496dfe843bbd7e7" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703" is_pre_election: true
I20240503 13:56:41.496615 15758 raft_consensus.cc:2454] T 37d81c890a1d411bb496dfe843bbd7e7 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 0.
I20240503 13:56:41.496670 13314 raft_consensus.cc:2454] T 37d81c890a1d411bb496dfe843bbd7e7 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 0.
I20240503 13:56:41.497009 15561 leader_election.cc:302] T 37d81c890a1d411bb496dfe843bbd7e7 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:41.497166 19586 raft_consensus.cc:2781] T 37d81c890a1d411bb496dfe843bbd7e7 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:41.497223 19586 raft_consensus.cc:482] T 37d81c890a1d411bb496dfe843bbd7e7 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:41.497270 19586 raft_consensus.cc:3037] T 37d81c890a1d411bb496dfe843bbd7e7 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:41.498261 19586 raft_consensus.cc:504] T 37d81c890a1d411bb496dfe843bbd7e7 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:41.498597 19586 leader_election.cc:288] T 37d81c890a1d411bb496dfe843bbd7e7 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 election: Requested vote from peers 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885), ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:41.498960 13314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37d81c890a1d411bb496dfe843bbd7e7" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703"
I20240503 13:56:41.499017 15758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37d81c890a1d411bb496dfe843bbd7e7" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2"
I20240503 13:56:41.499100 13314 raft_consensus.cc:3037] T 37d81c890a1d411bb496dfe843bbd7e7 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:41.499136 15758 raft_consensus.cc:3037] T 37d81c890a1d411bb496dfe843bbd7e7 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:41.500013 13314 raft_consensus.cc:2454] T 37d81c890a1d411bb496dfe843bbd7e7 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 1.
I20240503 13:56:41.500322 15758 raft_consensus.cc:2454] T 37d81c890a1d411bb496dfe843bbd7e7 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 1.
I20240503 13:56:41.500470 15561 leader_election.cc:302] T 37d81c890a1d411bb496dfe843bbd7e7 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:41.501070 19586 raft_consensus.cc:2781] T 37d81c890a1d411bb496dfe843bbd7e7 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:41.501242 19586 raft_consensus.cc:686] T 37d81c890a1d411bb496dfe843bbd7e7 P 009f849ee7b54c80a3f5264695047e4d [term 1 LEADER]: Becoming Leader. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Running, Role: LEADER
I20240503 13:56:41.501559 19586 consensus_queue.cc:237] T 37d81c890a1d411bb496dfe843bbd7e7 P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:41.502450 12911 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:41.502585 12911 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:41.503814 12909 catalog_manager.cc:5484] T 37d81c890a1d411bb496dfe843bbd7e7 P 009f849ee7b54c80a3f5264695047e4d reported cstate change: term changed from 0 to 1, leader changed from <none> to 009f849ee7b54c80a3f5264695047e4d (127.6.98.70). New cstate: current_term: 1 leader_uuid: "009f849ee7b54c80a3f5264695047e4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:41.505338 12911 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Sending DeleteTablet for 3 replicas of tablet d5c7e23182354fb792aa175f6dcb5f20
I20240503 13:56:41.505975 13032 tablet_service.cc:1514] Processing DeleteTablet for tablet d5c7e23182354fb792aa175f6dcb5f20 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:41 UTC) from {username='slave'} at 127.0.0.1:59458
I20240503 13:56:41.505970 15472 tablet_service.cc:1514] Processing DeleteTablet for tablet d5c7e23182354fb792aa175f6dcb5f20 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:41 UTC) from {username='slave'} at 127.0.0.1:58758
I20240503 13:56:41.506263 20527 tablet_replica.cc:330] T d5c7e23182354fb792aa175f6dcb5f20 P 61b887f0d40d497badeca5604c1f0546: stopping tablet replica
I20240503 13:56:41.506390 15867 tablet_service.cc:1514] Processing DeleteTablet for tablet d5c7e23182354fb792aa175f6dcb5f20 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:41 UTC) from {username='slave'} at 127.0.0.1:50498
I20240503 13:56:41.506429 20527 raft_consensus.cc:2229] T d5c7e23182354fb792aa175f6dcb5f20 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:41.506551 20527 raft_consensus.cc:2258] T d5c7e23182354fb792aa175f6dcb5f20 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:41.506757 20528 tablet_replica.cc:330] T d5c7e23182354fb792aa175f6dcb5f20 P 60ba3ffe3b7746469834cd91c84be14e: stopping tablet replica
I20240503 13:56:41.506896 20528 raft_consensus.cc:2229] T d5c7e23182354fb792aa175f6dcb5f20 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:41.507014 20528 raft_consensus.cc:2258] T d5c7e23182354fb792aa175f6dcb5f20 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:41.507254 20529 tablet_replica.cc:330] T d5c7e23182354fb792aa175f6dcb5f20 P e47105a0f46e4511b7112e571688a4c7: stopping tablet replica
I20240503 13:56:41.507396 20529 raft_consensus.cc:2229] T d5c7e23182354fb792aa175f6dcb5f20 P e47105a0f46e4511b7112e571688a4c7 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:56:41.507516 20527 ts_tablet_manager.cc:1875] T d5c7e23182354fb792aa175f6dcb5f20 P 61b887f0d40d497badeca5604c1f0546: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:41.507591 20529 raft_consensus.cc:2258] T d5c7e23182354fb792aa175f6dcb5f20 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:41.507799 20528 ts_tablet_manager.cc:1875] T d5c7e23182354fb792aa175f6dcb5f20 P 60ba3ffe3b7746469834cd91c84be14e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:41.508059 20529 ts_tablet_manager.cc:1875] T d5c7e23182354fb792aa175f6dcb5f20 P e47105a0f46e4511b7112e571688a4c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:41.510100 20527 ts_tablet_manager.cc:1888] T d5c7e23182354fb792aa175f6dcb5f20 P 61b887f0d40d497badeca5604c1f0546: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:41.510154 20529 ts_tablet_manager.cc:1888] T d5c7e23182354fb792aa175f6dcb5f20 P e47105a0f46e4511b7112e571688a4c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:41.510211 20529 log.cc:1196] T d5c7e23182354fb792aa175f6dcb5f20 P e47105a0f46e4511b7112e571688a4c7: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-7/wal/wals/d5c7e23182354fb792aa175f6dcb5f20
I20240503 13:56:41.510226 20527 log.cc:1196] T d5c7e23182354fb792aa175f6dcb5f20 P 61b887f0d40d497badeca5604c1f0546: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/wal/wals/d5c7e23182354fb792aa175f6dcb5f20
I20240503 13:56:41.510491 20527 ts_tablet_manager.cc:1909] T d5c7e23182354fb792aa175f6dcb5f20 P 61b887f0d40d497badeca5604c1f0546: Deleting consensus metadata
I20240503 13:56:41.510491 20529 ts_tablet_manager.cc:1909] T d5c7e23182354fb792aa175f6dcb5f20 P e47105a0f46e4511b7112e571688a4c7: Deleting consensus metadata
I20240503 13:56:41.511030 20528 ts_tablet_manager.cc:1888] T d5c7e23182354fb792aa175f6dcb5f20 P 60ba3ffe3b7746469834cd91c84be14e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:41.511183 12899 catalog_manager.cc:4830] TS 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153): tablet d5c7e23182354fb792aa175f6dcb5f20 (table rebalancer_dynamic_table_new_name [id=2a01aba75336445ca17e5ef317d14b36]) successfully deleted
I20240503 13:56:41.511235 20528 log.cc:1196] T d5c7e23182354fb792aa175f6dcb5f20 P 60ba3ffe3b7746469834cd91c84be14e: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal/wals/d5c7e23182354fb792aa175f6dcb5f20
I20240503 13:56:41.511197 12896 catalog_manager.cc:4830] TS e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): tablet d5c7e23182354fb792aa175f6dcb5f20 (table rebalancer_dynamic_table_new_name [id=2a01aba75336445ca17e5ef317d14b36]) successfully deleted
I20240503 13:56:41.511708 20528 ts_tablet_manager.cc:1909] T d5c7e23182354fb792aa175f6dcb5f20 P 60ba3ffe3b7746469834cd91c84be14e: Deleting consensus metadata
I20240503 13:56:41.512424 12897 catalog_manager.cc:4830] TS 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695): tablet d5c7e23182354fb792aa175f6dcb5f20 (table rebalancer_dynamic_table_new_name [id=2a01aba75336445ca17e5ef317d14b36]) successfully deleted
I20240503 13:56:41.534512 13424 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:56:41.535140 15736 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:56:41.539659 13185 raft_consensus.cc:1203] T c8a53b7095e948f988c9151882bfa1bd P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:56:41.546659 15605 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 13:56:41.548669 13032 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:41.549737 15472 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 13:56:41.551769 13164 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 13:56:41.555668 19586 raft_consensus.cc:482] T 3be039d9c2294eb5bec575fc0505280c P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:41.555776 19586 raft_consensus.cc:504] T 3be039d9c2294eb5bec575fc0505280c P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:41.556181 19586 leader_election.cc:288] T 3be039d9c2294eb5bec575fc0505280c P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153)
I20240503 13:56:41.560664 15888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3be039d9c2294eb5bec575fc0505280c" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7" is_pre_election: true
I20240503 13:56:41.560871 15888 raft_consensus.cc:2379] T 3be039d9c2294eb5bec575fc0505280c P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 009f849ee7b54c80a3f5264695047e4d in current term 1: Already voted for candidate 61b887f0d40d497badeca5604c1f0546 in this term.
I20240503 13:56:41.565387 15492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3be039d9c2294eb5bec575fc0505280c" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546" is_pre_election: true
I20240503 13:56:41.565784 15561 leader_election.cc:302] T 3be039d9c2294eb5bec575fc0505280c P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d; no voters: 61b887f0d40d497badeca5604c1f0546, e47105a0f46e4511b7112e571688a4c7
I20240503 13:56:41.566139 19586 raft_consensus.cc:3037] T 3be039d9c2294eb5bec575fc0505280c P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:41.566975 19586 raft_consensus.cc:2726] T 3be039d9c2294eb5bec575fc0505280c P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 13:56:41.585510 13296 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 13:56:41.588028 15624 raft_consensus.cc:1203] T b1395d66d8e04ecdafd7a30c50aa1c9a P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:56:41.592432 15867 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 13:56:41.607019 12909 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:56:41.612965 13424 tablet_service.cc:1467] Processing CreateTablet for tablet b50fa1d48dda42bb84bb23d3b27c0709 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=088520c06583457abb1f46e9ba282ec6]), partition=
I20240503 13:56:41.612931 15605 tablet_service.cc:1467] Processing CreateTablet for tablet b50fa1d48dda42bb84bb23d3b27c0709 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=088520c06583457abb1f46e9ba282ec6]), partition=
I20240503 13:56:41.613006 15472 tablet_service.cc:1467] Processing CreateTablet for tablet b50fa1d48dda42bb84bb23d3b27c0709 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=088520c06583457abb1f46e9ba282ec6]), partition=
I20240503 13:56:41.613229 15605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b50fa1d48dda42bb84bb23d3b27c0709. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:41.613229 13424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b50fa1d48dda42bb84bb23d3b27c0709. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:41.613250 15472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b50fa1d48dda42bb84bb23d3b27c0709. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:41.615451 20541 tablet_bootstrap.cc:492] T b50fa1d48dda42bb84bb23d3b27c0709 P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:41.615643 20333 tablet_bootstrap.cc:492] T b50fa1d48dda42bb84bb23d3b27c0709 P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:41.616652 20541 tablet_bootstrap.cc:654] T b50fa1d48dda42bb84bb23d3b27c0709 P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:41.616653 20333 tablet_bootstrap.cc:654] T b50fa1d48dda42bb84bb23d3b27c0709 P 61b887f0d40d497badeca5604c1f0546: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:41.617280 20448 tablet_bootstrap.cc:492] T b50fa1d48dda42bb84bb23d3b27c0709 P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:41.618132 20448 tablet_bootstrap.cc:654] T b50fa1d48dda42bb84bb23d3b27c0709 P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:41.619279 20448 tablet_bootstrap.cc:492] T b50fa1d48dda42bb84bb23d3b27c0709 P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:41.619366 20448 ts_tablet_manager.cc:1367] T b50fa1d48dda42bb84bb23d3b27c0709 P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:56:41.619845 20448 raft_consensus.cc:348] T b50fa1d48dda42bb84bb23d3b27c0709 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:41.619974 20448 raft_consensus.cc:374] T b50fa1d48dda42bb84bb23d3b27c0709 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:41.620007 20448 raft_consensus.cc:729] T b50fa1d48dda42bb84bb23d3b27c0709 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:41.620107 20448 consensus_queue.cc:260] T b50fa1d48dda42bb84bb23d3b27c0709 P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:41.620399 20448 ts_tablet_manager.cc:1398] T b50fa1d48dda42bb84bb23d3b27c0709 P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:41.622988 20333 tablet_bootstrap.cc:492] T b50fa1d48dda42bb84bb23d3b27c0709 P 61b887f0d40d497badeca5604c1f0546: No bootstrap required, opened a new log
I20240503 13:56:41.623097 20333 ts_tablet_manager.cc:1367] T b50fa1d48dda42bb84bb23d3b27c0709 P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.001s
I20240503 13:56:41.623584 20333 raft_consensus.cc:348] T b50fa1d48dda42bb84bb23d3b27c0709 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:41.623701 20333 raft_consensus.cc:374] T b50fa1d48dda42bb84bb23d3b27c0709 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:41.623726 20333 raft_consensus.cc:729] T b50fa1d48dda42bb84bb23d3b27c0709 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:41.623843 20333 consensus_queue.cc:260] T b50fa1d48dda42bb84bb23d3b27c0709 P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:41.624091 20333 ts_tablet_manager.cc:1398] T b50fa1d48dda42bb84bb23d3b27c0709 P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:41.644661 20541 tablet_bootstrap.cc:492] T b50fa1d48dda42bb84bb23d3b27c0709 P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:41.644781 20541 ts_tablet_manager.cc:1367] T b50fa1d48dda42bb84bb23d3b27c0709 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.029s user 0.000s sys 0.001s
I20240503 13:56:41.645284 20541 raft_consensus.cc:348] T b50fa1d48dda42bb84bb23d3b27c0709 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:41.645411 20541 raft_consensus.cc:374] T b50fa1d48dda42bb84bb23d3b27c0709 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:41.645439 20541 raft_consensus.cc:729] T b50fa1d48dda42bb84bb23d3b27c0709 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:41.645550 20541 consensus_queue.cc:260] T b50fa1d48dda42bb84bb23d3b27c0709 P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:41.645803 20541 ts_tablet_manager.cc:1398] T b50fa1d48dda42bb84bb23d3b27c0709 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:41.719885 20346 consensus_queue.cc:1035] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:41.738063 19828 consensus_queue.cc:1035] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:41.801245 19841 consensus_queue.cc:1035] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:41.804869 19618 consensus_queue.cc:1035] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:41.848660 19586 raft_consensus.cc:482] T b50fa1d48dda42bb84bb23d3b27c0709 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:41.848784 19586 raft_consensus.cc:504] T b50fa1d48dda42bb84bb23d3b27c0709 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:41.849184 19586 leader_election.cc:288] T b50fa1d48dda42bb84bb23d3b27c0709 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153)
I20240503 13:56:41.849561 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b50fa1d48dda42bb84bb23d3b27c0709" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
I20240503 13:56:41.849577 15492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b50fa1d48dda42bb84bb23d3b27c0709" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546" is_pre_election: true
I20240503 13:56:41.849691 13447 raft_consensus.cc:2454] T b50fa1d48dda42bb84bb23d3b27c0709 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 0.
I20240503 13:56:41.849711 15492 raft_consensus.cc:2454] T b50fa1d48dda42bb84bb23d3b27c0709 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 0.
I20240503 13:56:41.849964 15561 leader_election.cc:302] T b50fa1d48dda42bb84bb23d3b27c0709 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 61b887f0d40d497badeca5604c1f0546; no voters:
I20240503 13:56:41.850119 19586 raft_consensus.cc:2781] T b50fa1d48dda42bb84bb23d3b27c0709 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:41.850172 19586 raft_consensus.cc:482] T b50fa1d48dda42bb84bb23d3b27c0709 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:41.850207 19586 raft_consensus.cc:3037] T b50fa1d48dda42bb84bb23d3b27c0709 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:41.850993 19586 raft_consensus.cc:504] T b50fa1d48dda42bb84bb23d3b27c0709 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:41.851336 19586 leader_election.cc:288] T b50fa1d48dda42bb84bb23d3b27c0709 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 election: Requested vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153)
I20240503 13:56:41.851615 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b50fa1d48dda42bb84bb23d3b27c0709" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
I20240503 13:56:41.851730 13447 raft_consensus.cc:3037] T b50fa1d48dda42bb84bb23d3b27c0709 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:41.851842 15492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b50fa1d48dda42bb84bb23d3b27c0709" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546"
I20240503 13:56:41.851953 15492 raft_consensus.cc:3037] T b50fa1d48dda42bb84bb23d3b27c0709 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:41.852833 13447 raft_consensus.cc:2454] T b50fa1d48dda42bb84bb23d3b27c0709 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 1.
I20240503 13:56:41.852952 15492 raft_consensus.cc:2454] T b50fa1d48dda42bb84bb23d3b27c0709 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 1.
I20240503 13:56:41.853088 15558 leader_election.cc:302] T b50fa1d48dda42bb84bb23d3b27c0709 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 7690ee79577c45e3a2c8f8b552c0bff8; no voters:
I20240503 13:56:41.853255 20524 raft_consensus.cc:2781] T b50fa1d48dda42bb84bb23d3b27c0709 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:41.853317 20524 raft_consensus.cc:686] T b50fa1d48dda42bb84bb23d3b27c0709 P 009f849ee7b54c80a3f5264695047e4d [term 1 LEADER]: Becoming Leader. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Running, Role: LEADER
I20240503 13:56:41.853452 20524 consensus_queue.cc:237] T b50fa1d48dda42bb84bb23d3b27c0709 P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:41.855362 12903 catalog_manager.cc:5484] T b50fa1d48dda42bb84bb23d3b27c0709 P 009f849ee7b54c80a3f5264695047e4d reported cstate change: term changed from 0 to 1, leader changed from <none> to 009f849ee7b54c80a3f5264695047e4d (127.6.98.70). New cstate: current_term: 1 leader_uuid: "009f849ee7b54c80a3f5264695047e4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:41.861938 15758 raft_consensus.cc:1203] T eb92a057165f4da7b96d910d847ff761 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:56:41.876113 13424 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:56:41.877104 15472 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 13:56:41.878165 20524 consensus_queue.cc:1035] T 37d81c890a1d411bb496dfe843bbd7e7 P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:41.884367 15736 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:56:41.885807 19586 consensus_queue.cc:1035] T 37d81c890a1d411bb496dfe843bbd7e7 P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:41.887739 13032 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:41.887800 15605 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 13:56:41.889147 15867 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 13:56:41.890990 13296 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 13:56:41.925967 13164 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 13:56:41.965293 12903 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:56:41.969525 13447 raft_consensus.cc:1261] T b50fa1d48dda42bb84bb23d3b27c0709 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer 009f849ee7b54c80a3f5264695047e4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:41.969630 15492 raft_consensus.cc:1261] T b50fa1d48dda42bb84bb23d3b27c0709 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Refusing update from remote peer 009f849ee7b54c80a3f5264695047e4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:41.970037 20524 consensus_queue.cc:1035] T b50fa1d48dda42bb84bb23d3b27c0709 P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:41.970219 19586 consensus_queue.cc:1035] T b50fa1d48dda42bb84bb23d3b27c0709 P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:41.974583 12907 catalog_manager.cc:5576] TS 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109) does not have the latest schema for tablet b50fa1d48dda42bb84bb23d3b27c0709 (table rebalancer_dynamic_table [id=088520c06583457abb1f46e9ba282ec6]). Expected version 1 got 0
I20240503 13:56:41.975219 12903 catalog_manager.cc:5576] TS 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153) does not have the latest schema for tablet b50fa1d48dda42bb84bb23d3b27c0709 (table rebalancer_dynamic_table [id=088520c06583457abb1f46e9ba282ec6]). Expected version 1 got 0
I20240503 13:56:41.976133 20626 tablet.cc:1722] T b50fa1d48dda42bb84bb23d3b27c0709 P 009f849ee7b54c80a3f5264695047e4d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:41.976522 20626 tablet.cc:1620] T b50fa1d48dda42bb84bb23d3b27c0709 P 009f849ee7b54c80a3f5264695047e4d: MemRowSet was empty: no flush needed.
I20240503 13:56:41.976467 20627 tablet.cc:1722] T b50fa1d48dda42bb84bb23d3b27c0709 P 7690ee79577c45e3a2c8f8b552c0bff8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:41.977034 12907 catalog_manager.cc:5576] TS 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109) does not have the latest schema for tablet b50fa1d48dda42bb84bb23d3b27c0709 (table rebalancer_dynamic_table [id=088520c06583457abb1f46e9ba282ec6]). Expected version 1 got 0
I20240503 13:56:41.977156 20627 tablet.cc:1620] T b50fa1d48dda42bb84bb23d3b27c0709 P 7690ee79577c45e3a2c8f8b552c0bff8: MemRowSet was empty: no flush needed.
I20240503 13:56:41.979712 12907 catalog_manager.cc:5576] TS 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153) does not have the latest schema for tablet b50fa1d48dda42bb84bb23d3b27c0709 (table rebalancer_dynamic_table [id=088520c06583457abb1f46e9ba282ec6]). Expected version 1 got 0
I20240503 13:56:41.982877 20629 tablet.cc:1722] T b50fa1d48dda42bb84bb23d3b27c0709 P 61b887f0d40d497badeca5604c1f0546: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:41.982908 12907 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=088520c06583457abb1f46e9ba282ec6] alter complete (version 1)
I20240503 13:56:41.983319 20629 tablet.cc:1620] T b50fa1d48dda42bb84bb23d3b27c0709 P 61b887f0d40d497badeca5604c1f0546: MemRowSet was empty: no flush needed.
I20240503 13:56:41.984949 15888 consensus_queue.cc:237] T eb92a057165f4da7b96d910d847ff761 P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:41.986382 15757 raft_consensus.cc:1261] T eb92a057165f4da7b96d910d847ff761 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Refusing update from remote peer e47105a0f46e4511b7112e571688a4c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:56:41.986429 15493 raft_consensus.cc:1261] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Refusing update from remote peer e47105a0f46e4511b7112e571688a4c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:56:41.986879 19351 consensus_queue.cc:1035] T eb92a057165f4da7b96d910d847ff761 P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:56:41.987169 19351 consensus_queue.cc:1035] T eb92a057165f4da7b96d910d847ff761 P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:56:41.988700 19351 raft_consensus.cc:2932] T eb92a057165f4da7b96d910d847ff761 P e47105a0f46e4511b7112e571688a4c7 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:41.989153 15757 raft_consensus.cc:2932] T eb92a057165f4da7b96d910d847ff761 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:41.989215 15493 raft_consensus.cc:2932] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:41.991443 12903 catalog_manager.cc:5484] T eb92a057165f4da7b96d910d847ff761 P e47105a0f46e4511b7112e571688a4c7 reported cstate change: config changed from index 2 to 3, VOTER 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66) evicted. New cstate: current_term: 1 leader_uuid: "e47105a0f46e4511b7112e571688a4c7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:41.994915 13164 tablet_service.cc:1514] Processing DeleteTablet for tablet eb92a057165f4da7b96d910d847ff761 with delete_type TABLET_DATA_TOMBSTONED (TS 8ccf5a4ac450414c9d3579bd5800b770 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:41668
I20240503 13:56:41.995393 20635 tablet_replica.cc:330] T eb92a057165f4da7b96d910d847ff761 P 8ccf5a4ac450414c9d3579bd5800b770: stopping tablet replica
I20240503 13:56:41.995558 20635 raft_consensus.cc:2229] T eb92a057165f4da7b96d910d847ff761 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:41.995702 20635 raft_consensus.cc:2258] T eb92a057165f4da7b96d910d847ff761 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:41.996407 20635 ts_tablet_manager.cc:1875] T eb92a057165f4da7b96d910d847ff761 P 8ccf5a4ac450414c9d3579bd5800b770: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:56:41.998545 20635 ts_tablet_manager.cc:1888] T eb92a057165f4da7b96d910d847ff761 P 8ccf5a4ac450414c9d3579bd5800b770: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 13:56:41.998617 20635 log.cc:1196] T eb92a057165f4da7b96d910d847ff761 P 8ccf5a4ac450414c9d3579bd5800b770: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/wal/wals/eb92a057165f4da7b96d910d847ff761
I20240503 13:56:41.999136 12899 catalog_manager.cc:4830] TS 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387): tablet eb92a057165f4da7b96d910d847ff761 (table rebalancer_extra_table_10 [id=658f6c781f344047b9273985ceb17229]) successfully deleted
I20240503 13:56:42.039319 12905 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_extra_table_34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:56:42.051343 13032 tablet_service.cc:1467] Processing CreateTablet for tablet e03711163c7840209507960cd63d4cf1 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=f9a67cf3b21149e8ae304744bfe07f89]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:42.051684 13032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e03711163c7840209507960cd63d4cf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:42.054152 15867 tablet_service.cc:1467] Processing CreateTablet for tablet bc1a69158cd14828864659dd55ae9aff (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=f9a67cf3b21149e8ae304744bfe07f89]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:42.054354 13424 tablet_service.cc:1467] Processing CreateTablet for tablet bc1a69158cd14828864659dd55ae9aff (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=f9a67cf3b21149e8ae304744bfe07f89]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:42.054440 15867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc1a69158cd14828864659dd55ae9aff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:42.054621 13424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc1a69158cd14828864659dd55ae9aff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:42.055119 15868 tablet_service.cc:1467] Processing CreateTablet for tablet 324217b2f526432a85caa80935941130 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=f9a67cf3b21149e8ae304744bfe07f89]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:42.055325 15868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 324217b2f526432a85caa80935941130. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:42.055006 20676 tablet_bootstrap.cc:492] T e03711163c7840209507960cd63d4cf1 P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:42.056701 20541 tablet_bootstrap.cc:492] T bc1a69158cd14828864659dd55ae9aff P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:42.057261 20676 tablet_bootstrap.cc:654] T e03711163c7840209507960cd63d4cf1 P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:42.057386 13424 tablet_service.cc:1467] Processing CreateTablet for tablet 324217b2f526432a85caa80935941130 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=f9a67cf3b21149e8ae304744bfe07f89]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:42.057585 20541 tablet_bootstrap.cc:654] T bc1a69158cd14828864659dd55ae9aff P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:42.057616 13424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 324217b2f526432a85caa80935941130. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:42.058240 20677 tablet_bootstrap.cc:492] T bc1a69158cd14828864659dd55ae9aff P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:42.058434 20676 tablet_bootstrap.cc:492] T e03711163c7840209507960cd63d4cf1 P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:42.058537 20676 ts_tablet_manager.cc:1367] T e03711163c7840209507960cd63d4cf1 P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:56:42.058661 13164 tablet_service.cc:1467] Processing CreateTablet for tablet 324217b2f526432a85caa80935941130 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=f9a67cf3b21149e8ae304744bfe07f89]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:42.058931 13164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 324217b2f526432a85caa80935941130. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:42.058987 20676 raft_consensus.cc:348] T e03711163c7840209507960cd63d4cf1 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:42.059100 20677 tablet_bootstrap.cc:654] T bc1a69158cd14828864659dd55ae9aff P e47105a0f46e4511b7112e571688a4c7: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:42.059119 20676 raft_consensus.cc:374] T e03711163c7840209507960cd63d4cf1 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:42.059155 20676 raft_consensus.cc:729] T e03711163c7840209507960cd63d4cf1 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:42.059288 20676 consensus_queue.cc:260] T e03711163c7840209507960cd63d4cf1 P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:42.059602 20676 ts_tablet_manager.cc:1398] T e03711163c7840209507960cd63d4cf1 P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:42.061579 20677 tablet_bootstrap.cc:492] T bc1a69158cd14828864659dd55ae9aff P e47105a0f46e4511b7112e571688a4c7: No bootstrap required, opened a new log
I20240503 13:56:42.061702 15472 tablet_service.cc:1467] Processing CreateTablet for tablet e03711163c7840209507960cd63d4cf1 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=f9a67cf3b21149e8ae304744bfe07f89]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:42.061964 15472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e03711163c7840209507960cd63d4cf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:42.061988 15736 tablet_service.cc:1467] Processing CreateTablet for tablet bc1a69158cd14828864659dd55ae9aff (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=f9a67cf3b21149e8ae304744bfe07f89]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:42.062251 15736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc1a69158cd14828864659dd55ae9aff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:42.064288 20333 tablet_bootstrap.cc:492] T e03711163c7840209507960cd63d4cf1 P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:42.064790 15738 tablet_service.cc:1467] Processing CreateTablet for tablet e03711163c7840209507960cd63d4cf1 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=f9a67cf3b21149e8ae304744bfe07f89]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:42.065042 15738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e03711163c7840209507960cd63d4cf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:42.065271 20541 tablet_bootstrap.cc:492] T bc1a69158cd14828864659dd55ae9aff P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:42.065384 20333 tablet_bootstrap.cc:654] T e03711163c7840209507960cd63d4cf1 P 61b887f0d40d497badeca5604c1f0546: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:42.065382 20541 ts_tablet_manager.cc:1367] T bc1a69158cd14828864659dd55ae9aff P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.001s
I20240503 13:56:42.065932 20541 raft_consensus.cc:348] T bc1a69158cd14828864659dd55ae9aff P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:42.066052 20541 raft_consensus.cc:374] T bc1a69158cd14828864659dd55ae9aff P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:42.066088 20541 raft_consensus.cc:729] T bc1a69158cd14828864659dd55ae9aff P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:42.066263 20333 tablet_bootstrap.cc:492] T e03711163c7840209507960cd63d4cf1 P 61b887f0d40d497badeca5604c1f0546: No bootstrap required, opened a new log
I20240503 13:56:42.066211 20541 consensus_queue.cc:260] T bc1a69158cd14828864659dd55ae9aff P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:42.066360 20333 ts_tablet_manager.cc:1367] T e03711163c7840209507960cd63d4cf1 P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 13:56:42.066565 20541 ts_tablet_manager.cc:1398] T bc1a69158cd14828864659dd55ae9aff P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 13:56:42.066752 20333 raft_consensus.cc:348] T e03711163c7840209507960cd63d4cf1 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:42.066857 20333 raft_consensus.cc:374] T e03711163c7840209507960cd63d4cf1 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:42.066884 20333 raft_consensus.cc:729] T e03711163c7840209507960cd63d4cf1 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:42.067008 20333 consensus_queue.cc:260] T e03711163c7840209507960cd63d4cf1 P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:42.067260 20333 ts_tablet_manager.cc:1398] T e03711163c7840209507960cd63d4cf1 P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:42.061717 20677 ts_tablet_manager.cc:1367] T bc1a69158cd14828864659dd55ae9aff P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:56:42.067816 20503 raft_consensus.cc:482] T e03711163c7840209507960cd63d4cf1 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:42.067904 20503 raft_consensus.cc:504] T e03711163c7840209507960cd63d4cf1 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:42.068189 20677 raft_consensus.cc:348] T bc1a69158cd14828864659dd55ae9aff P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:42.068300 20677 raft_consensus.cc:374] T bc1a69158cd14828864659dd55ae9aff P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:42.068287 20503 leader_election.cc:288] T e03711163c7840209507960cd63d4cf1 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885), 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153)
I20240503 13:56:42.068336 20677 raft_consensus.cc:729] T bc1a69158cd14828864659dd55ae9aff P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:42.068478 20677 consensus_queue.cc:260] T bc1a69158cd14828864659dd55ae9aff P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:42.068982 15493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e03711163c7840209507960cd63d4cf1" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546" is_pre_election: true
I20240503 13:56:42.069090 15493 raft_consensus.cc:2454] T e03711163c7840209507960cd63d4cf1 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 0.
I20240503 13:56:42.069391 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e03711163c7840209507960cd63d4cf1" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2" is_pre_election: true
I20240503 13:56:42.069686 20677 ts_tablet_manager.cc:1398] T bc1a69158cd14828864659dd55ae9aff P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:42.069837 12990 leader_election.cc:302] T e03711163c7840209507960cd63d4cf1 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 61b887f0d40d497badeca5604c1f0546; no voters:
I20240503 13:56:42.069875 20677 tablet_bootstrap.cc:492] T 324217b2f526432a85caa80935941130 P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:42.070896 20677 tablet_bootstrap.cc:654] T 324217b2f526432a85caa80935941130 P e47105a0f46e4511b7112e571688a4c7: Neither blocks nor log segments found. Creating new log.
W20240503 13:56:42.070878 12990 leader_election.cc:341] T e03711163c7840209507960cd63d4cf1 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885): Illegal state: must be running to vote when last-logged opid is not known
I20240503 13:56:42.066758 20541 tablet_bootstrap.cc:492] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:42.071188 20503 raft_consensus.cc:2781] T e03711163c7840209507960cd63d4cf1 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:42.071249 20503 raft_consensus.cc:482] T e03711163c7840209507960cd63d4cf1 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:42.071285 20503 raft_consensus.cc:3037] T e03711163c7840209507960cd63d4cf1 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:42.071612 20681 tablet_bootstrap.cc:492] T 324217b2f526432a85caa80935941130 P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:42.072062 20541 tablet_bootstrap.cc:654] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:42.072360 20503 raft_consensus.cc:504] T e03711163c7840209507960cd63d4cf1 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:42.072831 20681 tablet_bootstrap.cc:654] T 324217b2f526432a85caa80935941130 P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:42.072880 20503 leader_election.cc:288] T e03711163c7840209507960cd63d4cf1 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 election: Requested vote from peers 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885), 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153)
I20240503 13:56:42.073098 20682 tablet_bootstrap.cc:492] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:42.073462 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e03711163c7840209507960cd63d4cf1" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2"
I20240503 13:56:42.074122 20682 tablet_bootstrap.cc:654] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:42.074193 15493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e03711163c7840209507960cd63d4cf1" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546"
I20240503 13:56:42.074304 15493 raft_consensus.cc:3037] T e03711163c7840209507960cd63d4cf1 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:42.075287 15493 raft_consensus.cc:2454] T e03711163c7840209507960cd63d4cf1 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 1.
I20240503 13:56:42.075649 12990 leader_election.cc:302] T e03711163c7840209507960cd63d4cf1 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 61b887f0d40d497badeca5604c1f0546; no voters:
W20240503 13:56:42.075804 12990 leader_election.cc:341] T e03711163c7840209507960cd63d4cf1 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885): Illegal state: must be running to vote when last-logged opid is not known
I20240503 13:56:42.076272 20503 raft_consensus.cc:2781] T e03711163c7840209507960cd63d4cf1 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:42.076357 20503 raft_consensus.cc:686] T e03711163c7840209507960cd63d4cf1 P 60ba3ffe3b7746469834cd91c84be14e [term 1 LEADER]: Becoming Leader. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Running, Role: LEADER
I20240503 13:56:42.076509 20503 consensus_queue.cc:237] T e03711163c7840209507960cd63d4cf1 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:42.076637 20681 tablet_bootstrap.cc:492] T 324217b2f526432a85caa80935941130 P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:42.076728 20681 ts_tablet_manager.cc:1367] T 324217b2f526432a85caa80935941130 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:56:42.077222 20681 raft_consensus.cc:348] T 324217b2f526432a85caa80935941130 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:42.077317 20541 tablet_bootstrap.cc:492] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:42.077334 20681 raft_consensus.cc:374] T 324217b2f526432a85caa80935941130 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:42.077368 20681 raft_consensus.cc:729] T 324217b2f526432a85caa80935941130 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:42.077414 20541 ts_tablet_manager.cc:1367] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 13:56:42.077502 20681 consensus_queue.cc:260] T 324217b2f526432a85caa80935941130 P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:42.077805 20681 ts_tablet_manager.cc:1398] T 324217b2f526432a85caa80935941130 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:42.077880 20541 raft_consensus.cc:348] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:42.077983 20541 raft_consensus.cc:374] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:42.078018 20541 raft_consensus.cc:729] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:42.078142 20541 consensus_queue.cc:260] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:42.078445 20541 ts_tablet_manager.cc:1398] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:42.079041 20677 tablet_bootstrap.cc:492] T 324217b2f526432a85caa80935941130 P e47105a0f46e4511b7112e571688a4c7: No bootstrap required, opened a new log
I20240503 13:56:42.079129 20677 ts_tablet_manager.cc:1367] T 324217b2f526432a85caa80935941130 P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 13:56:42.079728 20677 raft_consensus.cc:348] T 324217b2f526432a85caa80935941130 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:42.079845 20677 raft_consensus.cc:374] T 324217b2f526432a85caa80935941130 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:42.079891 20677 raft_consensus.cc:729] T 324217b2f526432a85caa80935941130 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:42.080027 20677 consensus_queue.cc:260] T 324217b2f526432a85caa80935941130 P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:42.080312 20677 ts_tablet_manager.cc:1398] T 324217b2f526432a85caa80935941130 P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:42.081126 12911 catalog_manager.cc:5484] T e03711163c7840209507960cd63d4cf1 P 60ba3ffe3b7746469834cd91c84be14e reported cstate change: term changed from 0 to 1, leader changed from <none> to 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65). New cstate: current_term: 1 leader_uuid: "60ba3ffe3b7746469834cd91c84be14e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } }
I20240503 13:56:42.081487 20682 tablet_bootstrap.cc:492] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:42.081588 20682 ts_tablet_manager.cc:1367] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 13:56:42.081993 20682 raft_consensus.cc:348] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:42.082113 20682 raft_consensus.cc:374] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:42.082146 20682 raft_consensus.cc:729] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:42.082279 20682 consensus_queue.cc:260] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:42.082549 20682 ts_tablet_manager.cc:1398] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:42.082723 20682 tablet_bootstrap.cc:492] T e03711163c7840209507960cd63d4cf1 P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:42.083889 20682 tablet_bootstrap.cc:654] T e03711163c7840209507960cd63d4cf1 P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:42.088168 20682 tablet_bootstrap.cc:492] T e03711163c7840209507960cd63d4cf1 P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:42.088263 20682 ts_tablet_manager.cc:1367] T e03711163c7840209507960cd63d4cf1 P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 13:56:42.088735 20682 raft_consensus.cc:348] T e03711163c7840209507960cd63d4cf1 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:42.088847 20682 raft_consensus.cc:374] T e03711163c7840209507960cd63d4cf1 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:42.088874 20682 raft_consensus.cc:729] T e03711163c7840209507960cd63d4cf1 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:42.088981 20682 consensus_queue.cc:260] T e03711163c7840209507960cd63d4cf1 P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:42.089288 20682 ts_tablet_manager.cc:1398] T e03711163c7840209507960cd63d4cf1 P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:42.089763 12905 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:56:42.097779 20626 tablet.cc:1722] T b50fa1d48dda42bb84bb23d3b27c0709 P 009f849ee7b54c80a3f5264695047e4d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:42.098109 20626 tablet.cc:1620] T b50fa1d48dda42bb84bb23d3b27c0709 P 009f849ee7b54c80a3f5264695047e4d: MemRowSet was empty: no flush needed.
I20240503 13:56:42.099036 20627 tablet.cc:1722] T b50fa1d48dda42bb84bb23d3b27c0709 P 7690ee79577c45e3a2c8f8b552c0bff8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:42.099367 20627 tablet.cc:1620] T b50fa1d48dda42bb84bb23d3b27c0709 P 7690ee79577c45e3a2c8f8b552c0bff8: MemRowSet was empty: no flush needed.
I20240503 13:56:42.102818 20629 tablet.cc:1722] T b50fa1d48dda42bb84bb23d3b27c0709 P 61b887f0d40d497badeca5604c1f0546: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:42.103122 20629 tablet.cc:1620] T b50fa1d48dda42bb84bb23d3b27c0709 P 61b887f0d40d497badeca5604c1f0546: MemRowSet was empty: no flush needed.
I20240503 13:56:42.106886 12908 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=088520c06583457abb1f46e9ba282ec6] alter complete (version 2)
I20240503 13:56:42.113688 19618 raft_consensus.cc:482] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:42.113821 19618 raft_consensus.cc:504] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:42.114217 19618 leader_election.cc:288] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:42.115054 15888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc1a69158cd14828864659dd55ae9aff" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7" is_pre_election: true
I20240503 13:56:42.115197 15888 raft_consensus.cc:2454] T bc1a69158cd14828864659dd55ae9aff P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 0.
I20240503 13:56:42.116360 15691 leader_election.cc:302] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76c386bef61a4bd5a907266a5118ffe2, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:42.117352 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc1a69158cd14828864659dd55ae9aff" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
I20240503 13:56:42.117501 13445 raft_consensus.cc:2454] T bc1a69158cd14828864659dd55ae9aff P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 0.
I20240503 13:56:42.121308 19618 raft_consensus.cc:2781] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:42.121380 19618 raft_consensus.cc:482] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:42.121424 19618 raft_consensus.cc:3037] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:42.122516 19618 raft_consensus.cc:504] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:42.122818 19618 leader_election.cc:288] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 election: Requested vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:42.124471 15888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc1a69158cd14828864659dd55ae9aff" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7"
I20240503 13:56:42.124779 15888 raft_consensus.cc:3037] T bc1a69158cd14828864659dd55ae9aff P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:42.125703 15888 raft_consensus.cc:2454] T bc1a69158cd14828864659dd55ae9aff P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 1.
I20240503 13:56:42.126081 15691 leader_election.cc:302] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76c386bef61a4bd5a907266a5118ffe2, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:42.127327 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc1a69158cd14828864659dd55ae9aff" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
I20240503 13:56:42.127446 13445 raft_consensus.cc:3037] T bc1a69158cd14828864659dd55ae9aff P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:42.128193 13445 raft_consensus.cc:2454] T bc1a69158cd14828864659dd55ae9aff P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 1.
I20240503 13:56:42.128465 19618 raft_consensus.cc:2781] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:42.128600 19618 raft_consensus.cc:686] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 [term 1 LEADER]: Becoming Leader. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Running, Role: LEADER
I20240503 13:56:42.128809 19618 consensus_queue.cc:237] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:42.132050 12905 catalog_manager.cc:5484] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71). New cstate: current_term: 1 leader_uuid: "76c386bef61a4bd5a907266a5118ffe2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } }
I20240503 13:56:42.145340 20120 raft_consensus.cc:482] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:42.145466 20120 raft_consensus.cc:504] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:42.145885 20120 leader_election.cc:288] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:42.146567 13185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324217b2f526432a85caa80935941130" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770" is_pre_election: true
I20240503 13:56:42.146703 13185 raft_consensus.cc:2454] T 324217b2f526432a85caa80935941130 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 0.
I20240503 13:56:42.146792 15888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324217b2f526432a85caa80935941130" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7" is_pre_election: true
I20240503 13:56:42.146926 15888 raft_consensus.cc:2454] T 324217b2f526432a85caa80935941130 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 0.
I20240503 13:56:42.146984 13383 leader_election.cc:302] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, 8ccf5a4ac450414c9d3579bd5800b770; no voters:
I20240503 13:56:42.148196 20194 raft_consensus.cc:2781] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:42.148257 20194 raft_consensus.cc:482] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:42.148296 20194 raft_consensus.cc:3037] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:42.149590 20194 raft_consensus.cc:504] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:42.149899 20194 leader_election.cc:288] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 election: Requested vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:42.151149 13185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324217b2f526432a85caa80935941130" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770"
I20240503 13:56:42.151268 13185 raft_consensus.cc:3037] T 324217b2f526432a85caa80935941130 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:42.152159 13185 raft_consensus.cc:2454] T 324217b2f526432a85caa80935941130 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 1.
I20240503 13:56:42.152688 13383 leader_election.cc:302] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, 8ccf5a4ac450414c9d3579bd5800b770; no voters:
I20240503 13:56:42.152889 20194 raft_consensus.cc:2781] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:42.152987 20194 raft_consensus.cc:686] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 LEADER]: Becoming Leader. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Running, Role: LEADER
I20240503 13:56:42.153120 20194 consensus_queue.cc:237] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:42.153172 15888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324217b2f526432a85caa80935941130" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7"
I20240503 13:56:42.153272 15888 raft_consensus.cc:3037] T 324217b2f526432a85caa80935941130 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:42.154218 15888 raft_consensus.cc:2454] T 324217b2f526432a85caa80935941130 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 1.
I20240503 13:56:42.155910 15605 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 13:56:42.156180 12903 catalog_manager.cc:5484] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68). New cstate: current_term: 1 leader_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } }
I20240503 13:56:42.165550 13164 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 13:56:42.166100 13032 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:42.166764 15736 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:42.169747 15472 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 13:56:42.170967 13424 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:42.192667 15868 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 13:56:42.196303 13296 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 13:56:42.210023 12903 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:56:42.220750 20627 tablet.cc:1722] T b50fa1d48dda42bb84bb23d3b27c0709 P 7690ee79577c45e3a2c8f8b552c0bff8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:42.229212 20626 tablet.cc:1722] T b50fa1d48dda42bb84bb23d3b27c0709 P 009f849ee7b54c80a3f5264695047e4d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:42.233201 20629 tablet.cc:1722] T b50fa1d48dda42bb84bb23d3b27c0709 P 61b887f0d40d497badeca5604c1f0546: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:42.239281 12903 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table_new_name [id=088520c06583457abb1f46e9ba282ec6] alter complete (version 3)
I20240503 13:56:42.272445 13185 consensus_queue.cc:237] T b82134ddae61479eb2ced348d7558799 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:42.273677 13314 raft_consensus.cc:1261] T b82134ddae61479eb2ced348d7558799 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Refusing update from remote peer 8ccf5a4ac450414c9d3579bd5800b770: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:56:42.273720 15757 raft_consensus.cc:1261] T b82134ddae61479eb2ced348d7558799 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Refusing update from remote peer 8ccf5a4ac450414c9d3579bd5800b770: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:56:42.274050 19019 consensus_queue.cc:1035] T b82134ddae61479eb2ced348d7558799 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:42.274389 19998 consensus_queue.cc:1035] T b82134ddae61479eb2ced348d7558799 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:42.275868 19019 raft_consensus.cc:2932] T b82134ddae61479eb2ced348d7558799 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:42.276599 13314 raft_consensus.cc:2932] T b82134ddae61479eb2ced348d7558799 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:42.277328 15758 raft_consensus.cc:2932] T b82134ddae61479eb2ced348d7558799 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:42.279021 12903 catalog_manager.cc:5484] T b82134ddae61479eb2ced348d7558799 P 8ccf5a4ac450414c9d3579bd5800b770 reported cstate change: config changed from index 4 to 5, VOTER 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68) evicted. New cstate: current_term: 1 leader_uuid: "8ccf5a4ac450414c9d3579bd5800b770" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:42.280664 19618 raft_consensus.cc:482] T e03711163c7840209507960cd63d4cf1 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:42.280745 19618 raft_consensus.cc:504] T e03711163c7840209507960cd63d4cf1 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:42.281549 19618 leader_election.cc:288] T e03711163c7840209507960cd63d4cf1 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153), 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:42.282166 15493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e03711163c7840209507960cd63d4cf1" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546" is_pre_election: true
I20240503 13:56:42.282307 15493 raft_consensus.cc:2379] T e03711163c7840209507960cd63d4cf1 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 76c386bef61a4bd5a907266a5118ffe2 in current term 1: Already voted for candidate 60ba3ffe3b7746469834cd91c84be14e in this term.
I20240503 13:56:42.282269 13424 tablet_service.cc:1514] Processing DeleteTablet for tablet b82134ddae61479eb2ced348d7558799 with delete_type TABLET_DATA_TOMBSTONED (TS 7690ee79577c45e3a2c8f8b552c0bff8 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:33678
I20240503 13:56:42.282738 13052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e03711163c7840209507960cd63d4cf1" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e" is_pre_election: true
I20240503 13:56:42.283304 15692 leader_election.cc:302] T e03711163c7840209507960cd63d4cf1 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76c386bef61a4bd5a907266a5118ffe2; no voters: 60ba3ffe3b7746469834cd91c84be14e, 61b887f0d40d497badeca5604c1f0546
I20240503 13:56:42.283725 19618 raft_consensus.cc:3037] T e03711163c7840209507960cd63d4cf1 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:42.284654 19618 raft_consensus.cc:2726] T e03711163c7840209507960cd63d4cf1 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 13:56:42.284794 20741 tablet_replica.cc:330] T b82134ddae61479eb2ced348d7558799 P 7690ee79577c45e3a2c8f8b552c0bff8: stopping tablet replica
I20240503 13:56:42.284946 20741 raft_consensus.cc:2229] T b82134ddae61479eb2ced348d7558799 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:42.285100 20741 raft_consensus.cc:2258] T b82134ddae61479eb2ced348d7558799 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:42.285686 20741 ts_tablet_manager.cc:1875] T b82134ddae61479eb2ced348d7558799 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:56:42.287726 20741 ts_tablet_manager.cc:1888] T b82134ddae61479eb2ced348d7558799 P 7690ee79577c45e3a2c8f8b552c0bff8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 13:56:42.287796 20741 log.cc:1196] T b82134ddae61479eb2ced348d7558799 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal/wals/b82134ddae61479eb2ced348d7558799
I20240503 13:56:42.288280 12896 catalog_manager.cc:4830] TS 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109): tablet b82134ddae61479eb2ced348d7558799 (table rebalance_test_table_9 [id=2fd11e7ec03a4f6891fc18482c1728a6]) successfully deleted
I20240503 13:56:42.347694 12906 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:42.347860 12906 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:42.350265 12906 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Sending DeleteTablet for 3 replicas of tablet b50fa1d48dda42bb84bb23d3b27c0709
I20240503 13:56:42.350874 13424 tablet_service.cc:1514] Processing DeleteTablet for tablet b50fa1d48dda42bb84bb23d3b27c0709 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:42 UTC) from {username='slave'} at 127.0.0.1:33678
I20240503 13:56:42.351063 20741 tablet_replica.cc:330] T b50fa1d48dda42bb84bb23d3b27c0709 P 7690ee79577c45e3a2c8f8b552c0bff8: stopping tablet replica
I20240503 13:56:42.351199 20741 raft_consensus.cc:2229] T b50fa1d48dda42bb84bb23d3b27c0709 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:42.351318 20741 raft_consensus.cc:2258] T b50fa1d48dda42bb84bb23d3b27c0709 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:42.351475 15605 tablet_service.cc:1514] Processing DeleteTablet for tablet b50fa1d48dda42bb84bb23d3b27c0709 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:42 UTC) from {username='slave'} at 127.0.0.1:60718
I20240503 13:56:42.352519 20741 ts_tablet_manager.cc:1875] T b50fa1d48dda42bb84bb23d3b27c0709 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:42.353830 15472 tablet_service.cc:1514] Processing DeleteTablet for tablet b50fa1d48dda42bb84bb23d3b27c0709 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:42 UTC) from {username='slave'} at 127.0.0.1:58758
I20240503 13:56:42.354686 20741 ts_tablet_manager.cc:1888] T b50fa1d48dda42bb84bb23d3b27c0709 P 7690ee79577c45e3a2c8f8b552c0bff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:42.354736 20741 log.cc:1196] T b50fa1d48dda42bb84bb23d3b27c0709 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal/wals/b50fa1d48dda42bb84bb23d3b27c0709
I20240503 13:56:42.355739 20741 ts_tablet_manager.cc:1909] T b50fa1d48dda42bb84bb23d3b27c0709 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting consensus metadata
I20240503 13:56:42.356451 12896 catalog_manager.cc:4830] TS 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109): tablet b50fa1d48dda42bb84bb23d3b27c0709 (table rebalancer_dynamic_table_new_name [id=088520c06583457abb1f46e9ba282ec6]) successfully deleted
I20240503 13:56:42.358770 20783 tablet_replica.cc:330] T b50fa1d48dda42bb84bb23d3b27c0709 P 009f849ee7b54c80a3f5264695047e4d: stopping tablet replica
I20240503 13:56:42.358986 20783 raft_consensus.cc:2229] T b50fa1d48dda42bb84bb23d3b27c0709 P 009f849ee7b54c80a3f5264695047e4d [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:56:42.361008 20783 raft_consensus.cc:2258] T b50fa1d48dda42bb84bb23d3b27c0709 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:42.361744 20783 ts_tablet_manager.cc:1875] T b50fa1d48dda42bb84bb23d3b27c0709 P 009f849ee7b54c80a3f5264695047e4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:42.362318 20784 tablet_replica.cc:330] T b50fa1d48dda42bb84bb23d3b27c0709 P 61b887f0d40d497badeca5604c1f0546: stopping tablet replica
I20240503 13:56:42.362660 20784 raft_consensus.cc:2229] T b50fa1d48dda42bb84bb23d3b27c0709 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:42.363243 20784 raft_consensus.cc:2258] T b50fa1d48dda42bb84bb23d3b27c0709 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:42.363863 20784 ts_tablet_manager.cc:1875] T b50fa1d48dda42bb84bb23d3b27c0709 P 61b887f0d40d497badeca5604c1f0546: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:42.364662 20783 ts_tablet_manager.cc:1888] T b50fa1d48dda42bb84bb23d3b27c0709 P 009f849ee7b54c80a3f5264695047e4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:42.364724 20783 log.cc:1196] T b50fa1d48dda42bb84bb23d3b27c0709 P 009f849ee7b54c80a3f5264695047e4d: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/wal/wals/b50fa1d48dda42bb84bb23d3b27c0709
I20240503 13:56:42.364923 20783 ts_tablet_manager.cc:1909] T b50fa1d48dda42bb84bb23d3b27c0709 P 009f849ee7b54c80a3f5264695047e4d: Deleting consensus metadata
I20240503 13:56:42.365710 20784 ts_tablet_manager.cc:1888] T b50fa1d48dda42bb84bb23d3b27c0709 P 61b887f0d40d497badeca5604c1f0546: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:42.365733 12896 catalog_manager.cc:4830] TS 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951): tablet b50fa1d48dda42bb84bb23d3b27c0709 (table rebalancer_dynamic_table_new_name [id=088520c06583457abb1f46e9ba282ec6]) successfully deleted
I20240503 13:56:42.365784 20784 log.cc:1196] T b50fa1d48dda42bb84bb23d3b27c0709 P 61b887f0d40d497badeca5604c1f0546: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/wal/wals/b50fa1d48dda42bb84bb23d3b27c0709
I20240503 13:56:42.366020 20784 ts_tablet_manager.cc:1909] T b50fa1d48dda42bb84bb23d3b27c0709 P 61b887f0d40d497badeca5604c1f0546: Deleting consensus metadata
I20240503 13:56:42.379134 12899 catalog_manager.cc:4830] TS 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153): tablet b50fa1d48dda42bb84bb23d3b27c0709 (table rebalancer_dynamic_table_new_name [id=088520c06583457abb1f46e9ba282ec6]) successfully deleted
I20240503 13:56:42.389780 13032 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:42.394937 15472 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 13:56:42.395258 15605 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 13:56:42.407945 15868 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 13:56:42.413857 13164 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 13:56:42.414281 13424 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:42.417446 15736 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:42.434072 13296 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 13:56:42.453087 12903 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:56:42.460016 15472 tablet_service.cc:1467] Processing CreateTablet for tablet 0bdd8f08752c45e4822cb50ef046f97d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f0065da3b077440fa837a74a94a7d5cc]), partition=
I20240503 13:56:42.460073 15605 tablet_service.cc:1467] Processing CreateTablet for tablet 0bdd8f08752c45e4822cb50ef046f97d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f0065da3b077440fa837a74a94a7d5cc]), partition=
I20240503 13:56:42.460317 15605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bdd8f08752c45e4822cb50ef046f97d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:42.460320 15472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bdd8f08752c45e4822cb50ef046f97d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:42.460753 15868 tablet_service.cc:1467] Processing CreateTablet for tablet 0bdd8f08752c45e4822cb50ef046f97d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f0065da3b077440fa837a74a94a7d5cc]), partition=
I20240503 13:56:42.461112 15868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bdd8f08752c45e4822cb50ef046f97d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:42.463579 20677 tablet_bootstrap.cc:492] T 0bdd8f08752c45e4822cb50ef046f97d P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:42.464561 20333 tablet_bootstrap.cc:492] T 0bdd8f08752c45e4822cb50ef046f97d P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:42.464627 20677 tablet_bootstrap.cc:654] T 0bdd8f08752c45e4822cb50ef046f97d P e47105a0f46e4511b7112e571688a4c7: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:42.465641 20333 tablet_bootstrap.cc:654] T 0bdd8f08752c45e4822cb50ef046f97d P 61b887f0d40d497badeca5604c1f0546: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:42.466056 20677 tablet_bootstrap.cc:492] T 0bdd8f08752c45e4822cb50ef046f97d P e47105a0f46e4511b7112e571688a4c7: No bootstrap required, opened a new log
I20240503 13:56:42.466195 20677 ts_tablet_manager.cc:1367] T 0bdd8f08752c45e4822cb50ef046f97d P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 13:56:42.466732 20677 raft_consensus.cc:348] T 0bdd8f08752c45e4822cb50ef046f97d P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:42.466876 20677 raft_consensus.cc:374] T 0bdd8f08752c45e4822cb50ef046f97d P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:42.466915 20677 raft_consensus.cc:729] T 0bdd8f08752c45e4822cb50ef046f97d P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:42.467042 20677 consensus_queue.cc:260] T 0bdd8f08752c45e4822cb50ef046f97d P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:42.467375 20677 ts_tablet_manager.cc:1398] T 0bdd8f08752c45e4822cb50ef046f97d P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:42.468911 20333 tablet_bootstrap.cc:492] T 0bdd8f08752c45e4822cb50ef046f97d P 61b887f0d40d497badeca5604c1f0546: No bootstrap required, opened a new log
I20240503 13:56:42.469012 20333 ts_tablet_manager.cc:1367] T 0bdd8f08752c45e4822cb50ef046f97d P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:56:42.469466 20333 raft_consensus.cc:348] T 0bdd8f08752c45e4822cb50ef046f97d P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:42.469574 20333 raft_consensus.cc:374] T 0bdd8f08752c45e4822cb50ef046f97d P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:42.469599 20333 raft_consensus.cc:729] T 0bdd8f08752c45e4822cb50ef046f97d P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:42.469695 20796 tablet_bootstrap.cc:492] T 0bdd8f08752c45e4822cb50ef046f97d P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:42.470392 20333 consensus_queue.cc:260] T 0bdd8f08752c45e4822cb50ef046f97d P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:42.470721 20796 tablet_bootstrap.cc:654] T 0bdd8f08752c45e4822cb50ef046f97d P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:42.471624 20333 ts_tablet_manager.cc:1398] T 0bdd8f08752c45e4822cb50ef046f97d P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:56:42.473066 20796 tablet_bootstrap.cc:492] T 0bdd8f08752c45e4822cb50ef046f97d P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:42.473218 20796 ts_tablet_manager.cc:1367] T 0bdd8f08752c45e4822cb50ef046f97d P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:56:42.473798 20796 raft_consensus.cc:348] T 0bdd8f08752c45e4822cb50ef046f97d P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:42.473898 20796 raft_consensus.cc:374] T 0bdd8f08752c45e4822cb50ef046f97d P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:42.473927 20796 raft_consensus.cc:729] T 0bdd8f08752c45e4822cb50ef046f97d P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:42.474072 20796 consensus_queue.cc:260] T 0bdd8f08752c45e4822cb50ef046f97d P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:42.474416 20796 ts_tablet_manager.cc:1398] T 0bdd8f08752c45e4822cb50ef046f97d P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:42.500319 13185 consensus_queue.cc:237] T a49b6f596d724b35b757ee69fe15e5a5 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:42.501590 13314 raft_consensus.cc:1261] T a49b6f596d724b35b757ee69fe15e5a5 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Refusing update from remote peer 8ccf5a4ac450414c9d3579bd5800b770: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:56:42.501637 15493 raft_consensus.cc:1261] T a49b6f596d724b35b757ee69fe15e5a5 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Refusing update from remote peer 8ccf5a4ac450414c9d3579bd5800b770: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:56:42.501988 19998 consensus_queue.cc:1035] T a49b6f596d724b35b757ee69fe15e5a5 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:42.502187 19019 consensus_queue.cc:1035] T a49b6f596d724b35b757ee69fe15e5a5 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:42.504012 20818 raft_consensus.cc:2932] T a49b6f596d724b35b757ee69fe15e5a5 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:42.504360 15493 raft_consensus.cc:2932] T a49b6f596d724b35b757ee69fe15e5a5 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:42.504384 13314 raft_consensus.cc:2932] T a49b6f596d724b35b757ee69fe15e5a5 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:42.507341 12904 catalog_manager.cc:5484] T a49b6f596d724b35b757ee69fe15e5a5 P ba0194098aa84449ae17e8a0d4c43703 reported cstate change: config changed from index 4 to 5, VOTER 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65) evicted. New cstate: current_term: 1 leader_uuid: "8ccf5a4ac450414c9d3579bd5800b770" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:42.510885 13032 tablet_service.cc:1514] Processing DeleteTablet for tablet a49b6f596d724b35b757ee69fe15e5a5 with delete_type TABLET_DATA_TOMBSTONED (TS 60ba3ffe3b7746469834cd91c84be14e not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:59458
I20240503 13:56:42.512022 20824 tablet_replica.cc:330] T a49b6f596d724b35b757ee69fe15e5a5 P 60ba3ffe3b7746469834cd91c84be14e: stopping tablet replica
I20240503 13:56:42.512156 20824 raft_consensus.cc:2229] T a49b6f596d724b35b757ee69fe15e5a5 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:42.512310 20824 raft_consensus.cc:2258] T a49b6f596d724b35b757ee69fe15e5a5 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:42.513123 20824 ts_tablet_manager.cc:1875] T a49b6f596d724b35b757ee69fe15e5a5 P 60ba3ffe3b7746469834cd91c84be14e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:56:42.515275 20824 ts_tablet_manager.cc:1888] T a49b6f596d724b35b757ee69fe15e5a5 P 60ba3ffe3b7746469834cd91c84be14e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 13:56:42.515342 20824 log.cc:1196] T a49b6f596d724b35b757ee69fe15e5a5 P 60ba3ffe3b7746469834cd91c84be14e: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal/wals/a49b6f596d724b35b757ee69fe15e5a5
I20240503 13:56:42.515848 12897 catalog_manager.cc:4830] TS 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695): tablet a49b6f596d724b35b757ee69fe15e5a5 (table rebalance_test_table_3 [id=b600248fbc0b4c71b7c7cd8dd2844bbf]) successfully deleted
I20240503 13:56:42.530548 19618 consensus_queue.cc:1035] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:42.535010 19618 consensus_queue.cc:1035] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:42.538890 20120 consensus_queue.cc:1035] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:42.542418 20120 consensus_queue.cc:1035] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:42.560788 19351 raft_consensus.cc:482] T 0bdd8f08752c45e4822cb50ef046f97d P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:42.560918 19351 raft_consensus.cc:504] T 0bdd8f08752c45e4822cb50ef046f97d P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:42.561326 19351 leader_election.cc:288] T 0bdd8f08752c45e4822cb50ef046f97d P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153), 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951)
I20240503 13:56:42.562089 15493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bdd8f08752c45e4822cb50ef046f97d" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546" is_pre_election: true
I20240503 13:56:42.562228 15493 raft_consensus.cc:2454] T 0bdd8f08752c45e4822cb50ef046f97d P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 0.
I20240503 13:56:42.562590 15825 leader_election.cc:302] T 0bdd8f08752c45e4822cb50ef046f97d P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61b887f0d40d497badeca5604c1f0546, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:42.562881 15624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bdd8f08752c45e4822cb50ef046f97d" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d" is_pre_election: true
I20240503 13:56:42.563014 15624 raft_consensus.cc:2454] T 0bdd8f08752c45e4822cb50ef046f97d P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 0.
I20240503 13:56:42.563475 19351 raft_consensus.cc:2781] T 0bdd8f08752c45e4822cb50ef046f97d P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:42.563541 19351 raft_consensus.cc:482] T 0bdd8f08752c45e4822cb50ef046f97d P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:42.563580 19351 raft_consensus.cc:3037] T 0bdd8f08752c45e4822cb50ef046f97d P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:42.564785 19351 raft_consensus.cc:504] T 0bdd8f08752c45e4822cb50ef046f97d P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:42.565127 19351 leader_election.cc:288] T 0bdd8f08752c45e4822cb50ef046f97d P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 election: Requested vote from peers 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153), 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951)
I20240503 13:56:42.565702 15624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bdd8f08752c45e4822cb50ef046f97d" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d"
I20240503 13:56:42.565814 15624 raft_consensus.cc:3037] T 0bdd8f08752c45e4822cb50ef046f97d P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:42.565783 15493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bdd8f08752c45e4822cb50ef046f97d" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546"
I20240503 13:56:42.565881 15493 raft_consensus.cc:3037] T 0bdd8f08752c45e4822cb50ef046f97d P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:42.566766 15624 raft_consensus.cc:2454] T 0bdd8f08752c45e4822cb50ef046f97d P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 1.
I20240503 13:56:42.566766 15493 raft_consensus.cc:2454] T 0bdd8f08752c45e4822cb50ef046f97d P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 1.
I20240503 13:56:42.567052 15825 leader_election.cc:302] T 0bdd8f08752c45e4822cb50ef046f97d P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61b887f0d40d497badeca5604c1f0546, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:42.567395 20232 raft_consensus.cc:2781] T 0bdd8f08752c45e4822cb50ef046f97d P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:42.567467 20232 raft_consensus.cc:686] T 0bdd8f08752c45e4822cb50ef046f97d P e47105a0f46e4511b7112e571688a4c7 [term 1 LEADER]: Becoming Leader. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Running, Role: LEADER
I20240503 13:56:42.567610 20232 consensus_queue.cc:237] T 0bdd8f08752c45e4822cb50ef046f97d P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:42.569759 12910 catalog_manager.cc:5484] T 0bdd8f08752c45e4822cb50ef046f97d P e47105a0f46e4511b7112e571688a4c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e47105a0f46e4511b7112e571688a4c7 (127.6.98.72). New cstate: current_term: 1 leader_uuid: "e47105a0f46e4511b7112e571688a4c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:42.579213 20503 consensus_queue.cc:1035] T e03711163c7840209507960cd63d4cf1 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:42.586828 20503 consensus_queue.cc:1035] T e03711163c7840209507960cd63d4cf1 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:42.620286 13032 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:42.625941 15472 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 13:56:42.628286 13164 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 13:56:42.631073 15736 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:42.631633 15605 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 13:56:42.636106 15868 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:56:42.636687 13424 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:42.656565 13296 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 13:56:42.674566 12910 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_extra_table_36"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:56:42.681859 15868 tablet_service.cc:1467] Processing CreateTablet for tablet 8e06d93b10e34b4aa98899c2e9d6e16f (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=2d47e8401e8a4898b648cecd450aaa8a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:42.681943 15736 tablet_service.cc:1467] Processing CreateTablet for tablet 8e06d93b10e34b4aa98899c2e9d6e16f (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=2d47e8401e8a4898b648cecd450aaa8a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:42.682193 15868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e06d93b10e34b4aa98899c2e9d6e16f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:42.682235 15736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e06d93b10e34b4aa98899c2e9d6e16f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:42.684418 20677 tablet_bootstrap.cc:492] T 8e06d93b10e34b4aa98899c2e9d6e16f P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:42.685179 15605 tablet_service.cc:1467] Processing CreateTablet for tablet a9a1258c39a144149bf5d03e9950dc06 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=2d47e8401e8a4898b648cecd450aaa8a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:42.685407 20677 tablet_bootstrap.cc:654] T 8e06d93b10e34b4aa98899c2e9d6e16f P e47105a0f46e4511b7112e571688a4c7: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:42.685441 15605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9a1258c39a144149bf5d03e9950dc06. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:42.685405 15867 tablet_service.cc:1467] Processing CreateTablet for tablet a9a1258c39a144149bf5d03e9950dc06 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=2d47e8401e8a4898b648cecd450aaa8a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:42.686503 15867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9a1258c39a144149bf5d03e9950dc06. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:42.686666 15472 tablet_service.cc:1467] Processing CreateTablet for tablet db5568722b6344d18d6d6d271f05fc7a (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=2d47e8401e8a4898b648cecd450aaa8a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:42.686774 13296 tablet_service.cc:1467] Processing CreateTablet for tablet db5568722b6344d18d6d6d271f05fc7a (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=2d47e8401e8a4898b648cecd450aaa8a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:42.686935 15472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db5568722b6344d18d6d6d271f05fc7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:42.687016 13296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db5568722b6344d18d6d6d271f05fc7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:42.687707 20884 tablet_bootstrap.cc:492] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:42.688730 20884 tablet_bootstrap.cc:654] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:42.688755 20796 tablet_bootstrap.cc:492] T a9a1258c39a144149bf5d03e9950dc06 P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:42.689096 20333 tablet_bootstrap.cc:492] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:42.689630 20796 tablet_bootstrap.cc:654] T a9a1258c39a144149bf5d03e9950dc06 P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:42.689972 20333 tablet_bootstrap.cc:654] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:42.690763 13424 tablet_service.cc:1467] Processing CreateTablet for tablet 8e06d93b10e34b4aa98899c2e9d6e16f (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=2d47e8401e8a4898b648cecd450aaa8a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:42.691031 13424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e06d93b10e34b4aa98899c2e9d6e16f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:42.691597 20333 tablet_bootstrap.cc:492] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546: No bootstrap required, opened a new log
I20240503 13:56:42.691676 20333 ts_tablet_manager.cc:1367] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:56:42.691790 20889 tablet_bootstrap.cc:492] T db5568722b6344d18d6d6d271f05fc7a P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:42.692106 20333 raft_consensus.cc:348] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:42.692211 20333 raft_consensus.cc:374] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:42.692243 20333 raft_consensus.cc:729] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:42.692363 20333 consensus_queue.cc:260] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:42.692698 20333 ts_tablet_manager.cc:1398] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:42.692755 20889 tablet_bootstrap.cc:654] T db5568722b6344d18d6d6d271f05fc7a P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:42.693504 15868 tablet_service.cc:1467] Processing CreateTablet for tablet db5568722b6344d18d6d6d271f05fc7a (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=2d47e8401e8a4898b648cecd450aaa8a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:42.693743 15868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db5568722b6344d18d6d6d271f05fc7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:42.693954 15736 tablet_service.cc:1467] Processing CreateTablet for tablet a9a1258c39a144149bf5d03e9950dc06 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=2d47e8401e8a4898b648cecd450aaa8a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:42.694185 15736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9a1258c39a144149bf5d03e9950dc06. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:42.694231 20796 tablet_bootstrap.cc:492] T a9a1258c39a144149bf5d03e9950dc06 P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:42.694301 20796 ts_tablet_manager.cc:1367] T a9a1258c39a144149bf5d03e9950dc06 P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 13:56:42.694730 20796 raft_consensus.cc:348] T a9a1258c39a144149bf5d03e9950dc06 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:42.694819 20796 raft_consensus.cc:374] T a9a1258c39a144149bf5d03e9950dc06 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:42.694844 20796 raft_consensus.cc:729] T a9a1258c39a144149bf5d03e9950dc06 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:42.694948 20796 consensus_queue.cc:260] T a9a1258c39a144149bf5d03e9950dc06 P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:42.695246 20796 ts_tablet_manager.cc:1398] T a9a1258c39a144149bf5d03e9950dc06 P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:42.696568 12906 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:56:42.699072 20891 tablet_bootstrap.cc:492] T 8e06d93b10e34b4aa98899c2e9d6e16f P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:42.700253 20891 tablet_bootstrap.cc:654] T 8e06d93b10e34b4aa98899c2e9d6e16f P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:42.701208 20677 tablet_bootstrap.cc:492] T 8e06d93b10e34b4aa98899c2e9d6e16f P e47105a0f46e4511b7112e571688a4c7: No bootstrap required, opened a new log
I20240503 13:56:42.701296 20677 ts_tablet_manager.cc:1367] T 8e06d93b10e34b4aa98899c2e9d6e16f P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.001s
I20240503 13:56:42.701460 20891 tablet_bootstrap.cc:492] T 8e06d93b10e34b4aa98899c2e9d6e16f P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:42.701558 20891 ts_tablet_manager.cc:1367] T 8e06d93b10e34b4aa98899c2e9d6e16f P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:56:42.701720 20677 raft_consensus.cc:348] T 8e06d93b10e34b4aa98899c2e9d6e16f P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:42.701820 20677 raft_consensus.cc:374] T 8e06d93b10e34b4aa98899c2e9d6e16f P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:42.701846 20677 raft_consensus.cc:729] T 8e06d93b10e34b4aa98899c2e9d6e16f P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:42.701956 20677 consensus_queue.cc:260] T 8e06d93b10e34b4aa98899c2e9d6e16f P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:42.702024 20891 raft_consensus.cc:348] T 8e06d93b10e34b4aa98899c2e9d6e16f P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:42.702126 20891 raft_consensus.cc:374] T 8e06d93b10e34b4aa98899c2e9d6e16f P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:42.702158 20891 raft_consensus.cc:729] T 8e06d93b10e34b4aa98899c2e9d6e16f P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:42.702244 20677 ts_tablet_manager.cc:1398] T 8e06d93b10e34b4aa98899c2e9d6e16f P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:42.702265 20891 consensus_queue.cc:260] T 8e06d93b10e34b4aa98899c2e9d6e16f P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:42.702392 20677 tablet_bootstrap.cc:492] T a9a1258c39a144149bf5d03e9950dc06 P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:42.702522 20891 ts_tablet_manager.cc:1398] T 8e06d93b10e34b4aa98899c2e9d6e16f P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:42.703274 20677 tablet_bootstrap.cc:654] T a9a1258c39a144149bf5d03e9950dc06 P e47105a0f46e4511b7112e571688a4c7: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:42.706185 15493 raft_consensus.cc:1261] T 0bdd8f08752c45e4822cb50ef046f97d P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Refusing update from remote peer e47105a0f46e4511b7112e571688a4c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:42.706261 15624 raft_consensus.cc:1261] T 0bdd8f08752c45e4822cb50ef046f97d P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Refusing update from remote peer e47105a0f46e4511b7112e571688a4c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:42.706477 20889 tablet_bootstrap.cc:492] T db5568722b6344d18d6d6d271f05fc7a P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:42.706566 20889 ts_tablet_manager.cc:1367] T db5568722b6344d18d6d6d271f05fc7a P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240503 13:56:42.706605 20232 consensus_queue.cc:1035] T 0bdd8f08752c45e4822cb50ef046f97d P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:42.706880 19351 consensus_queue.cc:1035] T 0bdd8f08752c45e4822cb50ef046f97d P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:42.707043 20889 raft_consensus.cc:348] T db5568722b6344d18d6d6d271f05fc7a P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:42.707152 20889 raft_consensus.cc:374] T db5568722b6344d18d6d6d271f05fc7a P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:42.707178 20889 raft_consensus.cc:729] T db5568722b6344d18d6d6d271f05fc7a P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:42.707280 20889 consensus_queue.cc:260] T db5568722b6344d18d6d6d271f05fc7a P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:42.707969 20677 tablet_bootstrap.cc:492] T a9a1258c39a144149bf5d03e9950dc06 P e47105a0f46e4511b7112e571688a4c7: No bootstrap required, opened a new log
I20240503 13:56:42.708046 20677 ts_tablet_manager.cc:1367] T a9a1258c39a144149bf5d03e9950dc06 P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20240503 13:56:42.709460 20677 raft_consensus.cc:348] T a9a1258c39a144149bf5d03e9950dc06 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:42.709549 12906 catalog_manager.cc:5576] TS 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153) does not have the latest schema for tablet 0bdd8f08752c45e4822cb50ef046f97d (table rebalancer_dynamic_table [id=f0065da3b077440fa837a74a94a7d5cc]). Expected version 1 got 0
I20240503 13:56:42.709578 20677 raft_consensus.cc:374] T a9a1258c39a144149bf5d03e9950dc06 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:42.709612 20677 raft_consensus.cc:729] T a9a1258c39a144149bf5d03e9950dc06 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:42.709721 20677 consensus_queue.cc:260] T a9a1258c39a144149bf5d03e9950dc06 P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:42.709980 20677 ts_tablet_manager.cc:1398] T a9a1258c39a144149bf5d03e9950dc06 P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:42.709967 20629 tablet.cc:1722] T 0bdd8f08752c45e4822cb50ef046f97d P 61b887f0d40d497badeca5604c1f0546: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:42.710150 20677 tablet_bootstrap.cc:492] T db5568722b6344d18d6d6d271f05fc7a P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:42.710265 20629 tablet.cc:1620] T 0bdd8f08752c45e4822cb50ef046f97d P 61b887f0d40d497badeca5604c1f0546: MemRowSet was empty: no flush needed.
I20240503 13:56:42.711118 20677 tablet_bootstrap.cc:654] T db5568722b6344d18d6d6d271f05fc7a P e47105a0f46e4511b7112e571688a4c7: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:42.712873 20677 tablet_bootstrap.cc:492] T db5568722b6344d18d6d6d271f05fc7a P e47105a0f46e4511b7112e571688a4c7: No bootstrap required, opened a new log
I20240503 13:56:42.713986 20889 ts_tablet_manager.cc:1398] T db5568722b6344d18d6d6d271f05fc7a P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240503 13:56:42.714169 20677 ts_tablet_manager.cc:1367] T db5568722b6344d18d6d6d271f05fc7a P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:56:42.714689 20677 raft_consensus.cc:348] T db5568722b6344d18d6d6d271f05fc7a P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:42.714910 20677 raft_consensus.cc:374] T db5568722b6344d18d6d6d271f05fc7a P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:42.715622 20677 raft_consensus.cc:729] T db5568722b6344d18d6d6d271f05fc7a P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:42.715806 20677 consensus_queue.cc:260] T db5568722b6344d18d6d6d271f05fc7a P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:42.716399 20677 ts_tablet_manager.cc:1398] T db5568722b6344d18d6d6d271f05fc7a P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:56:42.717238 20896 tablet.cc:1722] T 0bdd8f08752c45e4822cb50ef046f97d P e47105a0f46e4511b7112e571688a4c7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:42.717645 20896 tablet.cc:1620] T 0bdd8f08752c45e4822cb50ef046f97d P e47105a0f46e4511b7112e571688a4c7: MemRowSet was empty: no flush needed.
I20240503 13:56:42.718691 12906 catalog_manager.cc:5576] TS 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951) does not have the latest schema for tablet 0bdd8f08752c45e4822cb50ef046f97d (table rebalancer_dynamic_table [id=f0065da3b077440fa837a74a94a7d5cc]). Expected version 1 got 0
I20240503 13:56:42.718864 20626 tablet.cc:1722] T 0bdd8f08752c45e4822cb50ef046f97d P 009f849ee7b54c80a3f5264695047e4d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:42.719152 20626 tablet.cc:1620] T 0bdd8f08752c45e4822cb50ef046f97d P 009f849ee7b54c80a3f5264695047e4d: MemRowSet was empty: no flush needed.
I20240503 13:56:42.720098 12903 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=f0065da3b077440fa837a74a94a7d5cc] alter complete (version 1)
I20240503 13:56:42.722679 20884 tablet_bootstrap.cc:492] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:42.722781 20884 ts_tablet_manager.cc:1367] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.035s user 0.002s sys 0.000s
I20240503 13:56:42.723289 20884 raft_consensus.cc:348] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:42.723414 20884 raft_consensus.cc:374] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:42.723445 20884 raft_consensus.cc:729] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:42.723575 20884 consensus_queue.cc:260] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:42.723836 20884 ts_tablet_manager.cc:1398] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:42.724007 20884 tablet_bootstrap.cc:492] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:42.732100 20884 tablet_bootstrap.cc:654] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:42.737128 20884 tablet_bootstrap.cc:492] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:42.737351 20884 ts_tablet_manager.cc:1367] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.001s
I20240503 13:56:42.737641 13185 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "477c33f203f649019eff2b28524fad90"
dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59906
I20240503 13:56:42.737813 13185 raft_consensus.cc:595] T 477c33f203f649019eff2b28524fad90 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 LEADER]: Received request to transfer leadership
I20240503 13:56:42.737921 20884 raft_consensus.cc:348] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:42.738027 20884 raft_consensus.cc:374] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:42.738058 20884 raft_consensus.cc:729] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:42.738171 20884 consensus_queue.cc:260] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:42.738582 20884 ts_tablet_manager.cc:1398] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:42.746351 20232 raft_consensus.cc:482] T db5568722b6344d18d6d6d271f05fc7a P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:42.746459 20232 raft_consensus.cc:504] T db5568722b6344d18d6d6d271f05fc7a P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:42.746834 20232 leader_election.cc:288] T db5568722b6344d18d6d6d271f05fc7a P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153), ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:42.747239 15493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5568722b6344d18d6d6d271f05fc7a" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546" is_pre_election: true
I20240503 13:56:42.747239 13314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5568722b6344d18d6d6d271f05fc7a" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703" is_pre_election: true
I20240503 13:56:42.747363 15493 raft_consensus.cc:2454] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 0.
I20240503 13:56:42.747363 13314 raft_consensus.cc:2454] T db5568722b6344d18d6d6d271f05fc7a P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 0.
I20240503 13:56:42.747658 15825 leader_election.cc:302] T db5568722b6344d18d6d6d271f05fc7a P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61b887f0d40d497badeca5604c1f0546, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:42.747829 20232 raft_consensus.cc:2781] T db5568722b6344d18d6d6d271f05fc7a P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:42.747892 20232 raft_consensus.cc:482] T db5568722b6344d18d6d6d271f05fc7a P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:42.747927 20232 raft_consensus.cc:3037] T db5568722b6344d18d6d6d271f05fc7a P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:42.748963 20232 raft_consensus.cc:504] T db5568722b6344d18d6d6d271f05fc7a P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:42.749310 20232 leader_election.cc:288] T db5568722b6344d18d6d6d271f05fc7a P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 election: Requested vote from peers 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153), ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:42.752990 13314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5568722b6344d18d6d6d271f05fc7a" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703"
I20240503 13:56:42.752995 15493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5568722b6344d18d6d6d271f05fc7a" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546"
I20240503 13:56:42.753152 15493 raft_consensus.cc:3037] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:42.753152 13314 raft_consensus.cc:3037] T db5568722b6344d18d6d6d271f05fc7a P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:42.754177 13314 raft_consensus.cc:2454] T db5568722b6344d18d6d6d271f05fc7a P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 1.
I20240503 13:56:42.754179 15493 raft_consensus.cc:2454] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 1.
I20240503 13:56:42.754570 15825 leader_election.cc:302] T db5568722b6344d18d6d6d271f05fc7a P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba0194098aa84449ae17e8a0d4c43703, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:42.754765 20232 raft_consensus.cc:2781] T db5568722b6344d18d6d6d271f05fc7a P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:42.754871 20232 raft_consensus.cc:686] T db5568722b6344d18d6d6d271f05fc7a P e47105a0f46e4511b7112e571688a4c7 [term 1 LEADER]: Becoming Leader. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Running, Role: LEADER
I20240503 13:56:42.755015 20232 consensus_queue.cc:237] T db5568722b6344d18d6d6d271f05fc7a P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:42.757593 12906 catalog_manager.cc:5484] T db5568722b6344d18d6d6d271f05fc7a P e47105a0f46e4511b7112e571688a4c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e47105a0f46e4511b7112e571688a4c7 (127.6.98.72). New cstate: current_term: 1 leader_uuid: "e47105a0f46e4511b7112e571688a4c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:42.824555 12903 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:56:42.827574 19618 raft_consensus.cc:482] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:42.827695 19618 raft_consensus.cc:504] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:42.828099 19618 leader_election.cc:288] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:42.828617 15888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e06d93b10e34b4aa98899c2e9d6e16f" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7" is_pre_election: true
I20240503 13:56:42.828747 15888 raft_consensus.cc:2454] T 8e06d93b10e34b4aa98899c2e9d6e16f P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 0.
I20240503 13:56:42.828802 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e06d93b10e34b4aa98899c2e9d6e16f" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
I20240503 13:56:42.828958 13445 raft_consensus.cc:2454] T 8e06d93b10e34b4aa98899c2e9d6e16f P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 0.
I20240503 13:56:42.829248 15691 leader_election.cc:302] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76c386bef61a4bd5a907266a5118ffe2, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:42.830054 19618 raft_consensus.cc:2781] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:42.830108 19618 raft_consensus.cc:482] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:42.830143 19618 raft_consensus.cc:3037] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:42.831251 19618 raft_consensus.cc:504] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:42.831611 19618 leader_election.cc:288] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 election: Requested vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:42.832460 15888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e06d93b10e34b4aa98899c2e9d6e16f" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7"
I20240503 13:56:42.832590 15888 raft_consensus.cc:3037] T 8e06d93b10e34b4aa98899c2e9d6e16f P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:42.832764 20896 tablet.cc:1722] T 0bdd8f08752c45e4822cb50ef046f97d P e47105a0f46e4511b7112e571688a4c7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:42.833043 20896 tablet.cc:1620] T 0bdd8f08752c45e4822cb50ef046f97d P e47105a0f46e4511b7112e571688a4c7: MemRowSet was empty: no flush needed.
I20240503 13:56:42.833205 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e06d93b10e34b4aa98899c2e9d6e16f" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
I20240503 13:56:42.833315 13445 raft_consensus.cc:3037] T 8e06d93b10e34b4aa98899c2e9d6e16f P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:42.833563 15888 raft_consensus.cc:2454] T 8e06d93b10e34b4aa98899c2e9d6e16f P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 1.
I20240503 13:56:42.833845 15691 leader_election.cc:302] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76c386bef61a4bd5a907266a5118ffe2, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:42.834339 13445 raft_consensus.cc:2454] T 8e06d93b10e34b4aa98899c2e9d6e16f P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 1.
I20240503 13:56:42.835079 20626 tablet.cc:1722] T 0bdd8f08752c45e4822cb50ef046f97d P 009f849ee7b54c80a3f5264695047e4d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:42.835376 20626 tablet.cc:1620] T 0bdd8f08752c45e4822cb50ef046f97d P 009f849ee7b54c80a3f5264695047e4d: MemRowSet was empty: no flush needed.
I20240503 13:56:42.837709 19618 raft_consensus.cc:2781] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:42.837806 19618 raft_consensus.cc:686] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 [term 1 LEADER]: Becoming Leader. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Running, Role: LEADER
I20240503 13:56:42.837970 19618 consensus_queue.cc:237] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:42.839391 20629 tablet.cc:1722] T 0bdd8f08752c45e4822cb50ef046f97d P 61b887f0d40d497badeca5604c1f0546: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:42.839704 20629 tablet.cc:1620] T 0bdd8f08752c45e4822cb50ef046f97d P 61b887f0d40d497badeca5604c1f0546: MemRowSet was empty: no flush needed.
I20240503 13:56:42.840351 12903 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=f0065da3b077440fa837a74a94a7d5cc] alter complete (version 2)
I20240503 13:56:42.840914 12910 catalog_manager.cc:5484] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71). New cstate: current_term: 1 leader_uuid: "76c386bef61a4bd5a907266a5118ffe2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } }
I20240503 13:56:42.863490 19841 raft_consensus.cc:482] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:42.863662 19841 raft_consensus.cc:504] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:42.864102 19841 leader_election.cc:288] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:42.864529 15624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9a1258c39a144149bf5d03e9950dc06" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d" is_pre_election: true
I20240503 13:56:42.864529 15888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9a1258c39a144149bf5d03e9950dc06" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7" is_pre_election: true
I20240503 13:56:42.864696 15888 raft_consensus.cc:2454] T a9a1258c39a144149bf5d03e9950dc06 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 0.
I20240503 13:56:42.864696 15624 raft_consensus.cc:2454] T a9a1258c39a144149bf5d03e9950dc06 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 0.
I20240503 13:56:42.865005 15691 leader_election.cc:302] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 76c386bef61a4bd5a907266a5118ffe2; no voters:
I20240503 13:56:42.865185 19841 raft_consensus.cc:2781] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:42.865245 19841 raft_consensus.cc:482] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:42.865280 19841 raft_consensus.cc:3037] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:42.866209 19841 raft_consensus.cc:504] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:42.866503 19841 leader_election.cc:288] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 election: Requested vote from peers 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:42.866881 15624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9a1258c39a144149bf5d03e9950dc06" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d"
I20240503 13:56:42.866951 15888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9a1258c39a144149bf5d03e9950dc06" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7"
I20240503 13:56:42.866997 15624 raft_consensus.cc:3037] T a9a1258c39a144149bf5d03e9950dc06 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:42.867084 15888 raft_consensus.cc:3037] T a9a1258c39a144149bf5d03e9950dc06 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:42.868027 15624 raft_consensus.cc:2454] T a9a1258c39a144149bf5d03e9950dc06 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 1.
I20240503 13:56:42.868081 15888 raft_consensus.cc:2454] T a9a1258c39a144149bf5d03e9950dc06 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 1.
I20240503 13:56:42.868324 15691 leader_election.cc:302] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 76c386bef61a4bd5a907266a5118ffe2; no voters:
I20240503 13:56:42.868624 19841 raft_consensus.cc:2781] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:42.868757 19841 raft_consensus.cc:686] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 LEADER]: Becoming Leader. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Running, Role: LEADER
I20240503 13:56:42.868901 19841 consensus_queue.cc:237] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:42.871136 12903 catalog_manager.cc:5484] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71). New cstate: current_term: 1 leader_uuid: "76c386bef61a4bd5a907266a5118ffe2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:42.895226 15605 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 13:56:42.895687 13032 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:42.900441 13424 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:42.904434 15736 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 13:56:42.904865 15867 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 13:56:42.908932 13296 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 13:56:42.915699 15472 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 13:56:42.917050 13164 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 13:56:42.943938 12903 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:56:42.954433 20896 tablet.cc:1722] T 0bdd8f08752c45e4822cb50ef046f97d P e47105a0f46e4511b7112e571688a4c7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:42.955582 20629 tablet.cc:1722] T 0bdd8f08752c45e4822cb50ef046f97d P 61b887f0d40d497badeca5604c1f0546: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:42.958954 20626 tablet.cc:1722] T 0bdd8f08752c45e4822cb50ef046f97d P 009f849ee7b54c80a3f5264695047e4d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:42.961823 12903 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table_new_name [id=f0065da3b077440fa837a74a94a7d5cc] alter complete (version 3)
I20240503 13:56:42.980234 19019 raft_consensus.cc:979] T 477c33f203f649019eff2b28524fad90 P 8ccf5a4ac450414c9d3579bd5800b770: : Instructing follower ba0194098aa84449ae17e8a0d4c43703 to start an election
I20240503 13:56:42.980379 20818 raft_consensus.cc:1067] T 477c33f203f649019eff2b28524fad90 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 LEADER]: Signalling peer ba0194098aa84449ae17e8a0d4c43703 to start an election
I20240503 13:56:42.980955 13314 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "477c33f203f649019eff2b28524fad90"
dest_uuid: "ba0194098aa84449ae17e8a0d4c43703"
from {username='slave'} at 127.6.98.66:39305
I20240503 13:56:42.981117 13314 raft_consensus.cc:482] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 13:56:42.981179 13314 raft_consensus.cc:3037] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:56:42.982050 13314 raft_consensus.cc:504] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:42.982458 13314 leader_election.cc:288] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 2 election: Requested vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:42.982983 15889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "477c33f203f649019eff2b28524fad90" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "e47105a0f46e4511b7112e571688a4c7"
I20240503 13:56:42.983026 13185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "477c33f203f649019eff2b28524fad90" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770"
I20240503 13:56:42.983120 13185 raft_consensus.cc:3032] T 477c33f203f649019eff2b28524fad90 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 13:56:42.983126 15889 raft_consensus.cc:3037] T 477c33f203f649019eff2b28524fad90 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:56:42.983157 13185 raft_consensus.cc:729] T 477c33f203f649019eff2b28524fad90 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Running, Role: LEADER
I20240503 13:56:42.983309 13185 consensus_queue.cc:260] T 477c33f203f649019eff2b28524fad90 P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:42.983357 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "477c33f203f649019eff2b28524fad90" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
I20240503 13:56:42.983479 13445 raft_consensus.cc:3037] T 477c33f203f649019eff2b28524fad90 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:56:42.983513 13185 raft_consensus.cc:3037] T 477c33f203f649019eff2b28524fad90 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:56:42.984167 15889 raft_consensus.cc:2454] T 477c33f203f649019eff2b28524fad90 P e47105a0f46e4511b7112e571688a4c7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 2.
I20240503 13:56:42.984452 13445 raft_consensus.cc:2454] T 477c33f203f649019eff2b28524fad90 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 2.
I20240503 13:56:42.984472 13185 raft_consensus.cc:2454] T 477c33f203f649019eff2b28524fad90 P 8ccf5a4ac450414c9d3579bd5800b770 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 2.
I20240503 13:56:42.984807 13249 leader_election.cc:302] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, ba0194098aa84449ae17e8a0d4c43703, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:42.985018 18914 raft_consensus.cc:2781] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:56:42.985108 18914 raft_consensus.cc:686] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 [term 2 LEADER]: Becoming Leader. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Running, Role: LEADER
I20240503 13:56:42.985241 18914 consensus_queue.cc:237] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:42.987569 12909 catalog_manager.cc:5484] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 reported cstate change: term changed from 1 to 2, leader changed from 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66) to ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67). New cstate: current_term: 2 leader_uuid: "ba0194098aa84449ae17e8a0d4c43703" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:43.063138 12906 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:43.063303 12906 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:43.065899 12906 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Sending DeleteTablet for 3 replicas of tablet 0bdd8f08752c45e4822cb50ef046f97d
I20240503 13:56:43.066695 15472 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bdd8f08752c45e4822cb50ef046f97d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:43 UTC) from {username='slave'} at 127.0.0.1:58758
I20240503 13:56:43.067138 15867 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bdd8f08752c45e4822cb50ef046f97d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:43 UTC) from {username='slave'} at 127.0.0.1:50498
I20240503 13:56:43.067150 15605 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bdd8f08752c45e4822cb50ef046f97d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:43 UTC) from {username='slave'} at 127.0.0.1:60718
I20240503 13:56:43.067884 21023 tablet_replica.cc:330] T 0bdd8f08752c45e4822cb50ef046f97d P e47105a0f46e4511b7112e571688a4c7: stopping tablet replica
I20240503 13:56:43.068058 21022 tablet_replica.cc:330] T 0bdd8f08752c45e4822cb50ef046f97d P 61b887f0d40d497badeca5604c1f0546: stopping tablet replica
I20240503 13:56:43.068120 21023 raft_consensus.cc:2229] T 0bdd8f08752c45e4822cb50ef046f97d P e47105a0f46e4511b7112e571688a4c7 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:56:43.068200 21022 raft_consensus.cc:2229] T 0bdd8f08752c45e4822cb50ef046f97d P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:43.068316 21022 raft_consensus.cc:2258] T 0bdd8f08752c45e4822cb50ef046f97d P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:43.069092 21022 ts_tablet_manager.cc:1875] T 0bdd8f08752c45e4822cb50ef046f97d P 61b887f0d40d497badeca5604c1f0546: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:43.068979 21023 raft_consensus.cc:2258] T 0bdd8f08752c45e4822cb50ef046f97d P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:43.069967 21023 ts_tablet_manager.cc:1875] T 0bdd8f08752c45e4822cb50ef046f97d P e47105a0f46e4511b7112e571688a4c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:43.076476 21022 ts_tablet_manager.cc:1888] T 0bdd8f08752c45e4822cb50ef046f97d P 61b887f0d40d497badeca5604c1f0546: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:43.076683 21022 log.cc:1196] T 0bdd8f08752c45e4822cb50ef046f97d P 61b887f0d40d497badeca5604c1f0546: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/wal/wals/0bdd8f08752c45e4822cb50ef046f97d
I20240503 13:56:43.076975 21022 ts_tablet_manager.cc:1909] T 0bdd8f08752c45e4822cb50ef046f97d P 61b887f0d40d497badeca5604c1f0546: Deleting consensus metadata
I20240503 13:56:43.080317 21023 ts_tablet_manager.cc:1888] T 0bdd8f08752c45e4822cb50ef046f97d P e47105a0f46e4511b7112e571688a4c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:43.080399 21023 log.cc:1196] T 0bdd8f08752c45e4822cb50ef046f97d P e47105a0f46e4511b7112e571688a4c7: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-7/wal/wals/0bdd8f08752c45e4822cb50ef046f97d
I20240503 13:56:43.080677 21023 ts_tablet_manager.cc:1909] T 0bdd8f08752c45e4822cb50ef046f97d P e47105a0f46e4511b7112e571688a4c7: Deleting consensus metadata
I20240503 13:56:43.080945 21024 tablet_replica.cc:330] T 0bdd8f08752c45e4822cb50ef046f97d P 009f849ee7b54c80a3f5264695047e4d: stopping tablet replica
I20240503 13:56:43.081157 21024 raft_consensus.cc:2229] T 0bdd8f08752c45e4822cb50ef046f97d P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:43.081350 21024 raft_consensus.cc:2258] T 0bdd8f08752c45e4822cb50ef046f97d P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:43.081794 12896 catalog_manager.cc:4830] TS e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): tablet 0bdd8f08752c45e4822cb50ef046f97d (table rebalancer_dynamic_table_new_name [id=f0065da3b077440fa837a74a94a7d5cc]) successfully deleted
I20240503 13:56:43.082131 21024 ts_tablet_manager.cc:1875] T 0bdd8f08752c45e4822cb50ef046f97d P 009f849ee7b54c80a3f5264695047e4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:43.083562 12899 catalog_manager.cc:4830] TS 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153): tablet 0bdd8f08752c45e4822cb50ef046f97d (table rebalancer_dynamic_table_new_name [id=f0065da3b077440fa837a74a94a7d5cc]) successfully deleted
I20240503 13:56:43.085057 21024 ts_tablet_manager.cc:1888] T 0bdd8f08752c45e4822cb50ef046f97d P 009f849ee7b54c80a3f5264695047e4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:43.085191 21024 log.cc:1196] T 0bdd8f08752c45e4822cb50ef046f97d P 009f849ee7b54c80a3f5264695047e4d: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/wal/wals/0bdd8f08752c45e4822cb50ef046f97d
I20240503 13:56:43.085888 21024 ts_tablet_manager.cc:1909] T 0bdd8f08752c45e4822cb50ef046f97d P 009f849ee7b54c80a3f5264695047e4d: Deleting consensus metadata
I20240503 13:56:43.086642 12896 catalog_manager.cc:4830] TS 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951): tablet 0bdd8f08752c45e4822cb50ef046f97d (table rebalancer_dynamic_table_new_name [id=f0065da3b077440fa837a74a94a7d5cc]) successfully deleted
I20240503 13:56:43.127466 15605 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 13:56:43.127822 13424 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:43.143577 15867 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:56:43.143837 13032 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:43.145339 13164 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 13:56:43.149353 15472 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 13:56:43.168673 12907 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:56:43.174543 13032 tablet_service.cc:1467] Processing CreateTablet for tablet b537b5d76f234a5bb58ee4a6b28108ad (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d8f2a259ec1043d99817c72d96665386]), partition=
I20240503 13:56:43.174611 15472 tablet_service.cc:1467] Processing CreateTablet for tablet b537b5d76f234a5bb58ee4a6b28108ad (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d8f2a259ec1043d99817c72d96665386]), partition=
I20240503 13:56:43.174770 15605 tablet_service.cc:1467] Processing CreateTablet for tablet b537b5d76f234a5bb58ee4a6b28108ad (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d8f2a259ec1043d99817c72d96665386]), partition=
I20240503 13:56:43.174844 13032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b537b5d76f234a5bb58ee4a6b28108ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:43.174911 15472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b537b5d76f234a5bb58ee4a6b28108ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:43.175004 15605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b537b5d76f234a5bb58ee4a6b28108ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:43.177110 20796 tablet_bootstrap.cc:492] T b537b5d76f234a5bb58ee4a6b28108ad P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:43.177196 20333 tablet_bootstrap.cc:492] T b537b5d76f234a5bb58ee4a6b28108ad P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:43.177984 20796 tablet_bootstrap.cc:654] T b537b5d76f234a5bb58ee4a6b28108ad P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:43.178122 20333 tablet_bootstrap.cc:654] T b537b5d76f234a5bb58ee4a6b28108ad P 61b887f0d40d497badeca5604c1f0546: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:43.179764 20796 tablet_bootstrap.cc:492] T b537b5d76f234a5bb58ee4a6b28108ad P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:43.179857 20796 ts_tablet_manager.cc:1367] T b537b5d76f234a5bb58ee4a6b28108ad P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:56:43.180315 20796 raft_consensus.cc:348] T b537b5d76f234a5bb58ee4a6b28108ad P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:43.180398 20796 raft_consensus.cc:374] T b537b5d76f234a5bb58ee4a6b28108ad P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:43.180418 20796 raft_consensus.cc:729] T b537b5d76f234a5bb58ee4a6b28108ad P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:43.180537 20796 consensus_queue.cc:260] T b537b5d76f234a5bb58ee4a6b28108ad P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:43.180794 20796 ts_tablet_manager.cc:1398] T b537b5d76f234a5bb58ee4a6b28108ad P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:43.189160 20333 tablet_bootstrap.cc:492] T b537b5d76f234a5bb58ee4a6b28108ad P 61b887f0d40d497badeca5604c1f0546: No bootstrap required, opened a new log
I20240503 13:56:43.189267 20333 ts_tablet_manager.cc:1367] T b537b5d76f234a5bb58ee4a6b28108ad P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.000s
I20240503 13:56:43.189715 20333 raft_consensus.cc:348] T b537b5d76f234a5bb58ee4a6b28108ad P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:43.189816 20333 raft_consensus.cc:374] T b537b5d76f234a5bb58ee4a6b28108ad P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:43.189846 20333 raft_consensus.cc:729] T b537b5d76f234a5bb58ee4a6b28108ad P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:43.189971 20333 consensus_queue.cc:260] T b537b5d76f234a5bb58ee4a6b28108ad P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:43.190261 20333 ts_tablet_manager.cc:1398] T b537b5d76f234a5bb58ee4a6b28108ad P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:43.200557 13296 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 13:56:43.202154 21037 tablet_bootstrap.cc:492] T b537b5d76f234a5bb58ee4a6b28108ad P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:43.203167 21037 tablet_bootstrap.cc:654] T b537b5d76f234a5bb58ee4a6b28108ad P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:43.206200 15736 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 13:56:43.213284 21037 tablet_bootstrap.cc:492] T b537b5d76f234a5bb58ee4a6b28108ad P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:43.217536 21037 ts_tablet_manager.cc:1367] T b537b5d76f234a5bb58ee4a6b28108ad P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240503 13:56:43.218214 21037 raft_consensus.cc:348] T b537b5d76f234a5bb58ee4a6b28108ad P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:43.218436 21037 raft_consensus.cc:374] T b537b5d76f234a5bb58ee4a6b28108ad P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:43.218516 21037 raft_consensus.cc:729] T b537b5d76f234a5bb58ee4a6b28108ad P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:43.218719 21037 consensus_queue.cc:260] T b537b5d76f234a5bb58ee4a6b28108ad P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:43.220013 21037 ts_tablet_manager.cc:1398] T b537b5d76f234a5bb58ee4a6b28108ad P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:43.224620 20941 consensus_queue.cc:1035] T db5568722b6344d18d6d6d271f05fc7a P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:43.233338 20941 consensus_queue.cc:1035] T db5568722b6344d18d6d6d271f05fc7a P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:43.257237 15493 consensus_queue.cc:237] T 66121c347e6c443dac7c5123826a7ae1 P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:43.258582 13185 raft_consensus.cc:1261] T 66121c347e6c443dac7c5123826a7ae1 P 8ccf5a4ac450414c9d3579bd5800b770 [term 2 FOLLOWER]: Refusing update from remote peer 61b887f0d40d497badeca5604c1f0546: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 13:56:43.258625 13314 raft_consensus.cc:1261] T 66121c347e6c443dac7c5123826a7ae1 P ba0194098aa84449ae17e8a0d4c43703 [term 2 FOLLOWER]: Refusing update from remote peer 61b887f0d40d497badeca5604c1f0546: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 13:56:43.259003 20722 consensus_queue.cc:1035] T 66121c347e6c443dac7c5123826a7ae1 P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:56:43.259285 20346 consensus_queue.cc:1035] T 66121c347e6c443dac7c5123826a7ae1 P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:56:43.260192 20722 raft_consensus.cc:482] T b537b5d76f234a5bb58ee4a6b28108ad P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:43.260283 20722 raft_consensus.cc:504] T b537b5d76f234a5bb58ee4a6b28108ad P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:43.262063 20722 leader_election.cc:288] T b537b5d76f234a5bb58ee4a6b28108ad P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951), 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:43.262488 13051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b537b5d76f234a5bb58ee4a6b28108ad" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e" is_pre_election: true
I20240503 13:56:43.262636 13051 raft_consensus.cc:2454] T b537b5d76f234a5bb58ee4a6b28108ad P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 0.
I20240503 13:56:43.262866 20722 raft_consensus.cc:2932] T 66121c347e6c443dac7c5123826a7ae1 P 61b887f0d40d497badeca5604c1f0546 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:43.263151 15624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b537b5d76f234a5bb58ee4a6b28108ad" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d" is_pre_election: true
I20240503 13:56:43.263319 15624 raft_consensus.cc:2454] T b537b5d76f234a5bb58ee4a6b28108ad P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 0.
I20240503 13:56:43.264405 15426 leader_election.cc:302] T b537b5d76f234a5bb58ee4a6b28108ad P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 61b887f0d40d497badeca5604c1f0546; no voters:
I20240503 13:56:43.265689 20722 raft_consensus.cc:2781] T b537b5d76f234a5bb58ee4a6b28108ad P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:43.265755 20722 raft_consensus.cc:482] T b537b5d76f234a5bb58ee4a6b28108ad P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:43.265792 20722 raft_consensus.cc:3037] T b537b5d76f234a5bb58ee4a6b28108ad P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:43.266847 20722 raft_consensus.cc:504] T b537b5d76f234a5bb58ee4a6b28108ad P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:43.267228 20722 leader_election.cc:288] T b537b5d76f234a5bb58ee4a6b28108ad P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 election: Requested vote from peers 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951), 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:43.267424 12907 catalog_manager.cc:5484] T 66121c347e6c443dac7c5123826a7ae1 P 61b887f0d40d497badeca5604c1f0546 reported cstate change: config changed from index 4 to 6, VOTER 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65) evicted. New cstate: current_term: 2 leader_uuid: "61b887f0d40d497badeca5604c1f0546" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:43.267705 13051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b537b5d76f234a5bb58ee4a6b28108ad" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e"
I20240503 13:56:43.267832 13051 raft_consensus.cc:3037] T b537b5d76f234a5bb58ee4a6b28108ad P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:43.268332 13185 raft_consensus.cc:2932] T 66121c347e6c443dac7c5123826a7ae1 P 8ccf5a4ac450414c9d3579bd5800b770 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:43.269054 13051 raft_consensus.cc:2454] T b537b5d76f234a5bb58ee4a6b28108ad P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 1.
I20240503 13:56:43.269047 15624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b537b5d76f234a5bb58ee4a6b28108ad" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d"
I20240503 13:56:43.269177 15624 raft_consensus.cc:3037] T b537b5d76f234a5bb58ee4a6b28108ad P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:43.269843 15427 leader_election.cc:302] T b537b5d76f234a5bb58ee4a6b28108ad P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 61b887f0d40d497badeca5604c1f0546; no voters:
I20240503 13:56:43.270231 15624 raft_consensus.cc:2454] T b537b5d76f234a5bb58ee4a6b28108ad P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 1.
I20240503 13:56:43.270612 13314 raft_consensus.cc:2932] T 66121c347e6c443dac7c5123826a7ae1 P ba0194098aa84449ae17e8a0d4c43703 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:43.271313 21062 raft_consensus.cc:2781] T b537b5d76f234a5bb58ee4a6b28108ad P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:43.271399 21062 raft_consensus.cc:686] T b537b5d76f234a5bb58ee4a6b28108ad P 61b887f0d40d497badeca5604c1f0546 [term 1 LEADER]: Becoming Leader. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Running, Role: LEADER
I20240503 13:56:43.271534 21062 consensus_queue.cc:237] T b537b5d76f234a5bb58ee4a6b28108ad P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:43.273597 13032 tablet_service.cc:1514] Processing DeleteTablet for tablet 66121c347e6c443dac7c5123826a7ae1 with delete_type TABLET_DATA_TOMBSTONED (TS 60ba3ffe3b7746469834cd91c84be14e not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59458
I20240503 13:56:43.275462 12911 catalog_manager.cc:5484] T b537b5d76f234a5bb58ee4a6b28108ad P 61b887f0d40d497badeca5604c1f0546 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61b887f0d40d497badeca5604c1f0546 (127.6.98.69). New cstate: current_term: 1 leader_uuid: "61b887f0d40d497badeca5604c1f0546" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:43.278192 21064 tablet_replica.cc:330] T 66121c347e6c443dac7c5123826a7ae1 P 60ba3ffe3b7746469834cd91c84be14e: stopping tablet replica
I20240503 13:56:43.278402 21064 raft_consensus.cc:2229] T 66121c347e6c443dac7c5123826a7ae1 P 60ba3ffe3b7746469834cd91c84be14e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:43.278527 21064 raft_consensus.cc:2258] T 66121c347e6c443dac7c5123826a7ae1 P 60ba3ffe3b7746469834cd91c84be14e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:43.279151 21064 ts_tablet_manager.cc:1875] T 66121c347e6c443dac7c5123826a7ae1 P 60ba3ffe3b7746469834cd91c84be14e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:56:43.281713 21064 ts_tablet_manager.cc:1888] T 66121c347e6c443dac7c5123826a7ae1 P 60ba3ffe3b7746469834cd91c84be14e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 13:56:43.281836 21064 log.cc:1196] T 66121c347e6c443dac7c5123826a7ae1 P 60ba3ffe3b7746469834cd91c84be14e: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal/wals/66121c347e6c443dac7c5123826a7ae1
I20240503 13:56:43.282441 12897 catalog_manager.cc:4830] TS 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695): tablet 66121c347e6c443dac7c5123826a7ae1 (table rebalance_test_table_11 [id=ba5c23cc430441c0a236a4359195fcb0]) successfully deleted
I20240503 13:56:43.305346 19841 consensus_queue.cc:1035] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:43.306872 19618 consensus_queue.cc:1035] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:43.308830 19618 consensus_queue.cc:1035] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:43.318709 19841 consensus_queue.cc:1035] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:43.393043 12904 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:56:43.397866 13051 raft_consensus.cc:1261] T b537b5d76f234a5bb58ee4a6b28108ad P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Refusing update from remote peer 61b887f0d40d497badeca5604c1f0546: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:43.398305 15624 raft_consensus.cc:1261] T b537b5d76f234a5bb58ee4a6b28108ad P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Refusing update from remote peer 61b887f0d40d497badeca5604c1f0546: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:43.398481 21062 consensus_queue.cc:1035] T b537b5d76f234a5bb58ee4a6b28108ad P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:43.399324 21062 consensus_queue.cc:1035] T b537b5d76f234a5bb58ee4a6b28108ad P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:43.401798 12911 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_extra_table_38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:56:43.402007 12903 catalog_manager.cc:5576] TS 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951) does not have the latest schema for tablet b537b5d76f234a5bb58ee4a6b28108ad (table rebalancer_dynamic_table [id=d8f2a259ec1043d99817c72d96665386]). Expected version 1 got 0
I20240503 13:56:43.404059 12904 catalog_manager.cc:5576] TS 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695) does not have the latest schema for tablet b537b5d76f234a5bb58ee4a6b28108ad (table rebalancer_dynamic_table [id=d8f2a259ec1043d99817c72d96665386]). Expected version 1 got 0
I20240503 13:56:43.409729 13032 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:43.410387 15605 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 13:56:43.411111 20629 tablet.cc:1722] T b537b5d76f234a5bb58ee4a6b28108ad P 61b887f0d40d497badeca5604c1f0546: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:43.411314 20587 mvcc.cc:205] Tried to move back new op lower bound from 7023593895513751552 to 7023593895001690112. Current Snapshot: MvccSnapshot[applied={T|T < 7023593895513751552}]
I20240503 13:56:43.411476 20629 tablet.cc:1620] T b537b5d76f234a5bb58ee4a6b28108ad P 61b887f0d40d497badeca5604c1f0546: MemRowSet was empty: no flush needed.
I20240503 13:56:43.411468 20626 tablet.cc:1722] T b537b5d76f234a5bb58ee4a6b28108ad P 009f849ee7b54c80a3f5264695047e4d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:43.411756 20626 tablet.cc:1620] T b537b5d76f234a5bb58ee4a6b28108ad P 009f849ee7b54c80a3f5264695047e4d: MemRowSet was empty: no flush needed.
I20240503 13:56:43.412278 13032 tablet_service.cc:1467] Processing CreateTablet for tablet 9dbcb7be2adb4655829c190c6cc6af85 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=a790c3879f9148469fcff4b1c82f6650]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:43.412577 13032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dbcb7be2adb4655829c190c6cc6af85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:43.412604 15736 tablet_service.cc:1467] Processing CreateTablet for tablet 5eeac82f64ed4e458ed86e452d70430d (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=a790c3879f9148469fcff4b1c82f6650]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:43.412889 15736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5eeac82f64ed4e458ed86e452d70430d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:43.413229 15867 tablet_service.cc:1467] Processing CreateTablet for tablet c2df7186d52a46fea36c672a22a1a0f0 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=a790c3879f9148469fcff4b1c82f6650]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:43.413501 15867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2df7186d52a46fea36c672a22a1a0f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:43.414783 21037 tablet_bootstrap.cc:492] T 9dbcb7be2adb4655829c190c6cc6af85 P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:43.414983 15604 tablet_service.cc:1467] Processing CreateTablet for tablet 5eeac82f64ed4e458ed86e452d70430d (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=a790c3879f9148469fcff4b1c82f6650]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:43.415197 15604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5eeac82f64ed4e458ed86e452d70430d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:43.415702 21037 tablet_bootstrap.cc:654] T 9dbcb7be2adb4655829c190c6cc6af85 P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:43.415760 21127 tablet_bootstrap.cc:492] T c2df7186d52a46fea36c672a22a1a0f0 P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:43.415920 15868 tablet_service.cc:1467] Processing CreateTablet for tablet 9dbcb7be2adb4655829c190c6cc6af85 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=a790c3879f9148469fcff4b1c82f6650]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:43.416129 15868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dbcb7be2adb4655829c190c6cc6af85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:43.416751 21127 tablet_bootstrap.cc:654] T c2df7186d52a46fea36c672a22a1a0f0 P e47105a0f46e4511b7112e571688a4c7: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:43.417312 20796 tablet_bootstrap.cc:492] T 5eeac82f64ed4e458ed86e452d70430d P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:43.418231 20796 tablet_bootstrap.cc:654] T 5eeac82f64ed4e458ed86e452d70430d P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:43.419862 13164 tablet_service.cc:1467] Processing CreateTablet for tablet 5eeac82f64ed4e458ed86e452d70430d (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=a790c3879f9148469fcff4b1c82f6650]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:43.420111 13164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5eeac82f64ed4e458ed86e452d70430d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:43.423892 15605 tablet_service.cc:1467] Processing CreateTablet for tablet c2df7186d52a46fea36c672a22a1a0f0 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=a790c3879f9148469fcff4b1c82f6650]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:43.424163 15605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2df7186d52a46fea36c672a22a1a0f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:43.428118 15472 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 13:56:43.429385 15472 tablet_service.cc:1467] Processing CreateTablet for tablet c2df7186d52a46fea36c672a22a1a0f0 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=a790c3879f9148469fcff4b1c82f6650]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:43.429637 15471 tablet_service.cc:1467] Processing CreateTablet for tablet 9dbcb7be2adb4655829c190c6cc6af85 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=a790c3879f9148469fcff4b1c82f6650]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:43.432518 13185 raft_consensus.cc:1261] T 477c33f203f649019eff2b28524fad90 P 8ccf5a4ac450414c9d3579bd5800b770 [term 2 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 13:56:43.434489 21126 tablet_bootstrap.cc:492] T 5eeac82f64ed4e458ed86e452d70430d P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:43.434409 20215 consensus_queue.cc:1035] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:43.437052 21132 tablet.cc:1722] T b537b5d76f234a5bb58ee4a6b28108ad P 60ba3ffe3b7746469834cd91c84be14e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:43.437566 21132 tablet.cc:1620] T b537b5d76f234a5bb58ee4a6b28108ad P 60ba3ffe3b7746469834cd91c84be14e: MemRowSet was empty: no flush needed.
I20240503 13:56:43.439673 12903 catalog_manager.cc:5576] TS 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695) does not have the latest schema for tablet b537b5d76f234a5bb58ee4a6b28108ad (table rebalancer_dynamic_table [id=d8f2a259ec1043d99817c72d96665386]). Expected version 1 got 0
I20240503 13:56:43.439750 13424 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:43.440049 15868 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:56:43.446166 12904 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=d8f2a259ec1043d99817c72d96665386] alter complete (version 1)
I20240503 13:56:43.429639 15472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2df7186d52a46fea36c672a22a1a0f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:43.450366 20333 tablet_bootstrap.cc:492] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:43.451339 20333 tablet_bootstrap.cc:654] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:43.452579 15471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dbcb7be2adb4655829c190c6cc6af85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:43.457700 15736 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 13:56:43.458029 13164 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 13:56:43.460961 13296 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 13:56:43.468068 20796 tablet_bootstrap.cc:492] T 5eeac82f64ed4e458ed86e452d70430d P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:43.468169 20796 ts_tablet_manager.cc:1367] T 5eeac82f64ed4e458ed86e452d70430d P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.051s user 0.002s sys 0.000s
I20240503 13:56:43.468763 20796 raft_consensus.cc:348] T 5eeac82f64ed4e458ed86e452d70430d P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:43.468874 20796 raft_consensus.cc:374] T 5eeac82f64ed4e458ed86e452d70430d P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:43.468901 20796 raft_consensus.cc:729] T 5eeac82f64ed4e458ed86e452d70430d P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:43.469009 20796 consensus_queue.cc:260] T 5eeac82f64ed4e458ed86e452d70430d P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:43.469261 20796 ts_tablet_manager.cc:1398] T 5eeac82f64ed4e458ed86e452d70430d P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:43.469424 20796 tablet_bootstrap.cc:492] T c2df7186d52a46fea36c672a22a1a0f0 P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:43.470350 20796 tablet_bootstrap.cc:654] T c2df7186d52a46fea36c672a22a1a0f0 P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:43.475191 20333 tablet_bootstrap.cc:492] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546: No bootstrap required, opened a new log
I20240503 13:56:43.475314 20333 ts_tablet_manager.cc:1367] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.000s
I20240503 13:56:43.475844 20333 raft_consensus.cc:348] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:43.475955 20333 raft_consensus.cc:374] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:43.475984 20333 raft_consensus.cc:729] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:43.476101 20333 consensus_queue.cc:260] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:43.476383 20333 ts_tablet_manager.cc:1398] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:43.476620 20333 tablet_bootstrap.cc:492] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:43.477618 20796 tablet_bootstrap.cc:492] T c2df7186d52a46fea36c672a22a1a0f0 P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:43.477638 20333 tablet_bootstrap.cc:654] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:43.477700 20796 ts_tablet_manager.cc:1367] T c2df7186d52a46fea36c672a22a1a0f0 P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 13:56:43.478111 20796 raft_consensus.cc:348] T c2df7186d52a46fea36c672a22a1a0f0 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:43.478204 20796 raft_consensus.cc:374] T c2df7186d52a46fea36c672a22a1a0f0 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:43.478227 20796 raft_consensus.cc:729] T c2df7186d52a46fea36c672a22a1a0f0 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:43.478327 20796 consensus_queue.cc:260] T c2df7186d52a46fea36c672a22a1a0f0 P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:43.478585 20796 ts_tablet_manager.cc:1398] T c2df7186d52a46fea36c672a22a1a0f0 P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:43.485025 21130 tablet_bootstrap.cc:492] T 5eeac82f64ed4e458ed86e452d70430d P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:43.487797 21130 tablet_bootstrap.cc:654] T 5eeac82f64ed4e458ed86e452d70430d P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:43.488946 21037 tablet_bootstrap.cc:492] T 9dbcb7be2adb4655829c190c6cc6af85 P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:43.489077 21037 ts_tablet_manager.cc:1367] T 9dbcb7be2adb4655829c190c6cc6af85 P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.074s user 0.002s sys 0.000s
I20240503 13:56:43.489560 21037 raft_consensus.cc:348] T 9dbcb7be2adb4655829c190c6cc6af85 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:43.489681 21037 raft_consensus.cc:374] T 9dbcb7be2adb4655829c190c6cc6af85 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:43.489707 21037 raft_consensus.cc:729] T 9dbcb7be2adb4655829c190c6cc6af85 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:43.489817 21037 consensus_queue.cc:260] T 9dbcb7be2adb4655829c190c6cc6af85 P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:43.490092 21037 ts_tablet_manager.cc:1398] T 9dbcb7be2adb4655829c190c6cc6af85 P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:43.492002 21126 tablet_bootstrap.cc:654] T 5eeac82f64ed4e458ed86e452d70430d P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:43.494153 21127 tablet_bootstrap.cc:492] T c2df7186d52a46fea36c672a22a1a0f0 P e47105a0f46e4511b7112e571688a4c7: No bootstrap required, opened a new log
I20240503 13:56:43.494256 21127 ts_tablet_manager.cc:1367] T c2df7186d52a46fea36c672a22a1a0f0 P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.079s user 0.002s sys 0.000s
I20240503 13:56:43.494709 21127 raft_consensus.cc:348] T c2df7186d52a46fea36c672a22a1a0f0 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:43.494840 21127 raft_consensus.cc:374] T c2df7186d52a46fea36c672a22a1a0f0 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:43.494879 21127 raft_consensus.cc:729] T c2df7186d52a46fea36c672a22a1a0f0 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:43.495000 21127 consensus_queue.cc:260] T c2df7186d52a46fea36c672a22a1a0f0 P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:43.495290 21127 ts_tablet_manager.cc:1398] T c2df7186d52a46fea36c672a22a1a0f0 P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:43.495440 21127 tablet_bootstrap.cc:492] T 9dbcb7be2adb4655829c190c6cc6af85 P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:43.496337 21127 tablet_bootstrap.cc:654] T 9dbcb7be2adb4655829c190c6cc6af85 P e47105a0f46e4511b7112e571688a4c7: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:43.500706 21127 tablet_bootstrap.cc:492] T 9dbcb7be2adb4655829c190c6cc6af85 P e47105a0f46e4511b7112e571688a4c7: No bootstrap required, opened a new log
I20240503 13:56:43.500810 21127 ts_tablet_manager.cc:1367] T 9dbcb7be2adb4655829c190c6cc6af85 P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:56:43.501267 21127 raft_consensus.cc:348] T 9dbcb7be2adb4655829c190c6cc6af85 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:43.501385 21127 raft_consensus.cc:374] T 9dbcb7be2adb4655829c190c6cc6af85 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:43.501585 21127 raft_consensus.cc:729] T 9dbcb7be2adb4655829c190c6cc6af85 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:43.501730 21127 consensus_queue.cc:260] T 9dbcb7be2adb4655829c190c6cc6af85 P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:43.501993 21127 ts_tablet_manager.cc:1398] T 9dbcb7be2adb4655829c190c6cc6af85 P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:43.508786 21130 tablet_bootstrap.cc:492] T 5eeac82f64ed4e458ed86e452d70430d P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:43.508900 21130 ts_tablet_manager.cc:1367] T 5eeac82f64ed4e458ed86e452d70430d P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.075s user 0.002s sys 0.000s
I20240503 13:56:43.509385 21130 raft_consensus.cc:348] T 5eeac82f64ed4e458ed86e452d70430d P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:43.509501 21130 raft_consensus.cc:374] T 5eeac82f64ed4e458ed86e452d70430d P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:43.509528 21130 raft_consensus.cc:729] T 5eeac82f64ed4e458ed86e452d70430d P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:43.509665 21130 consensus_queue.cc:260] T 5eeac82f64ed4e458ed86e452d70430d P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:43.509949 21130 ts_tablet_manager.cc:1398] T 5eeac82f64ed4e458ed86e452d70430d P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:43.510465 21126 tablet_bootstrap.cc:492] T 5eeac82f64ed4e458ed86e452d70430d P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:43.510562 21126 ts_tablet_manager.cc:1367] T 5eeac82f64ed4e458ed86e452d70430d P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.076s user 0.002s sys 0.000s
I20240503 13:56:43.510980 21126 raft_consensus.cc:348] T 5eeac82f64ed4e458ed86e452d70430d P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:43.511098 21126 raft_consensus.cc:374] T 5eeac82f64ed4e458ed86e452d70430d P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:43.511124 21126 raft_consensus.cc:729] T 5eeac82f64ed4e458ed86e452d70430d P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:43.511240 21126 consensus_queue.cc:260] T 5eeac82f64ed4e458ed86e452d70430d P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:43.511489 21126 ts_tablet_manager.cc:1398] T 5eeac82f64ed4e458ed86e452d70430d P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:43.512938 15889 raft_consensus.cc:1261] T 477c33f203f649019eff2b28524fad90 P e47105a0f46e4511b7112e571688a4c7 [term 2 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 13:56:43.513304 18914 consensus_queue.cc:1035] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:43.513384 20333 tablet_bootstrap.cc:492] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546: No bootstrap required, opened a new log
I20240503 13:56:43.513473 20333 ts_tablet_manager.cc:1367] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.037s user 0.002s sys 0.000s
I20240503 13:56:43.513980 20333 raft_consensus.cc:348] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:43.514097 20333 raft_consensus.cc:374] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:43.514125 20333 raft_consensus.cc:729] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:43.514232 20333 consensus_queue.cc:260] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:43.514503 20333 ts_tablet_manager.cc:1398] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:43.520941 13445 raft_consensus.cc:1261] T 477c33f203f649019eff2b28524fad90 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 2 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 13:56:43.521778 20215 consensus_queue.cc:1035] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:43.542557 13445 consensus_queue.cc:237] T b1395d66d8e04ecdafd7a30c50aa1c9a P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:43.543742 15624 raft_consensus.cc:1261] T b1395d66d8e04ecdafd7a30c50aa1c9a P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Refusing update from remote peer 7690ee79577c45e3a2c8f8b552c0bff8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:56:43.544322 21036 consensus_queue.cc:1035] T b1395d66d8e04ecdafd7a30c50aa1c9a P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:43.544580 20722 raft_consensus.cc:482] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:43.544631 13183 raft_consensus.cc:1261] T b1395d66d8e04ecdafd7a30c50aa1c9a P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Refusing update from remote peer 7690ee79577c45e3a2c8f8b552c0bff8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:56:43.544697 20722 raft_consensus.cc:504] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:43.545071 20722 leader_election.cc:288] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:43.545444 15889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dbcb7be2adb4655829c190c6cc6af85" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7" is_pre_election: true
I20240503 13:56:43.545593 15889 raft_consensus.cc:2454] T 9dbcb7be2adb4655829c190c6cc6af85 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 0.
I20240503 13:56:43.545626 13051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dbcb7be2adb4655829c190c6cc6af85" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e" is_pre_election: true
I20240503 13:56:43.545684 21036 consensus_queue.cc:1035] T b1395d66d8e04ecdafd7a30c50aa1c9a P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:43.545740 13051 raft_consensus.cc:2454] T 9dbcb7be2adb4655829c190c6cc6af85 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 0.
I20240503 13:56:43.546023 15427 leader_election.cc:302] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 61b887f0d40d497badeca5604c1f0546; no voters:
I20240503 13:56:43.546209 20722 raft_consensus.cc:2781] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:43.546273 20722 raft_consensus.cc:482] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:43.546319 20722 raft_consensus.cc:3037] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:43.547307 20722 raft_consensus.cc:504] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:43.547694 20722 leader_election.cc:288] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 election: Requested vote from peers 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:43.547966 13051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dbcb7be2adb4655829c190c6cc6af85" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e"
I20240503 13:56:43.548095 13051 raft_consensus.cc:3037] T 9dbcb7be2adb4655829c190c6cc6af85 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:43.549202 13051 raft_consensus.cc:2454] T 9dbcb7be2adb4655829c190c6cc6af85 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 1.
I20240503 13:56:43.549561 15427 leader_election.cc:302] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 61b887f0d40d497badeca5604c1f0546; no voters:
I20240503 13:56:43.549830 21062 raft_consensus.cc:2781] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:43.549903 21062 raft_consensus.cc:686] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 [term 1 LEADER]: Becoming Leader. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Running, Role: LEADER
I20240503 13:56:43.550050 21062 consensus_queue.cc:237] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:43.550436 15889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dbcb7be2adb4655829c190c6cc6af85" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7"
I20240503 13:56:43.550544 15889 raft_consensus.cc:3037] T 9dbcb7be2adb4655829c190c6cc6af85 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:43.551477 15889 raft_consensus.cc:2454] T 9dbcb7be2adb4655829c190c6cc6af85 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 1.
I20240503 13:56:43.552749 21036 raft_consensus.cc:2932] T b1395d66d8e04ecdafd7a30c50aa1c9a P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:43.553624 12906 catalog_manager.cc:5484] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61b887f0d40d497badeca5604c1f0546 (127.6.98.69). New cstate: current_term: 1 leader_uuid: "61b887f0d40d497badeca5604c1f0546" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:43.554648 15624 raft_consensus.cc:2932] T b1395d66d8e04ecdafd7a30c50aa1c9a P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:43.556208 12911 catalog_manager.cc:5484] T b1395d66d8e04ecdafd7a30c50aa1c9a P 7690ee79577c45e3a2c8f8b552c0bff8 reported cstate change: config changed from index 4 to 5, VOTER ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) evicted. New cstate: current_term: 1 leader_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:43.556429 12903 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:56:43.563095 13296 tablet_service.cc:1514] Processing DeleteTablet for tablet b1395d66d8e04ecdafd7a30c50aa1c9a with delete_type TABLET_DATA_TOMBSTONED (TS ba0194098aa84449ae17e8a0d4c43703 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:33250
I20240503 13:56:43.563537 13185 raft_consensus.cc:2932] T b1395d66d8e04ecdafd7a30c50aa1c9a P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:43.567289 20629 tablet.cc:1722] T b537b5d76f234a5bb58ee4a6b28108ad P 61b887f0d40d497badeca5604c1f0546: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:43.567375 20626 tablet.cc:1722] T b537b5d76f234a5bb58ee4a6b28108ad P 009f849ee7b54c80a3f5264695047e4d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:43.567631 20629 tablet.cc:1620] T b537b5d76f234a5bb58ee4a6b28108ad P 61b887f0d40d497badeca5604c1f0546: MemRowSet was empty: no flush needed.
I20240503 13:56:43.567629 20626 tablet.cc:1620] T b537b5d76f234a5bb58ee4a6b28108ad P 009f849ee7b54c80a3f5264695047e4d: MemRowSet was empty: no flush needed.
I20240503 13:56:43.568105 21168 tablet_replica.cc:330] T b1395d66d8e04ecdafd7a30c50aa1c9a P ba0194098aa84449ae17e8a0d4c43703: stopping tablet replica
I20240503 13:56:43.568243 21168 raft_consensus.cc:2229] T b1395d66d8e04ecdafd7a30c50aa1c9a P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:43.568434 21168 raft_consensus.cc:2258] T b1395d66d8e04ecdafd7a30c50aa1c9a P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:43.569010 21168 ts_tablet_manager.cc:1875] T b1395d66d8e04ecdafd7a30c50aa1c9a P ba0194098aa84449ae17e8a0d4c43703: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:56:43.571535 21132 tablet.cc:1722] T b537b5d76f234a5bb58ee4a6b28108ad P 60ba3ffe3b7746469834cd91c84be14e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:43.571862 21132 tablet.cc:1620] T b537b5d76f234a5bb58ee4a6b28108ad P 60ba3ffe3b7746469834cd91c84be14e: MemRowSet was empty: no flush needed.
I20240503 13:56:43.573091 21168 ts_tablet_manager.cc:1888] T b1395d66d8e04ecdafd7a30c50aa1c9a P ba0194098aa84449ae17e8a0d4c43703: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 13:56:43.573210 21168 log.cc:1196] T b1395d66d8e04ecdafd7a30c50aa1c9a P ba0194098aa84449ae17e8a0d4c43703: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/wal/wals/b1395d66d8e04ecdafd7a30c50aa1c9a
I20240503 13:56:43.573769 12899 catalog_manager.cc:4830] TS ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279): tablet b1395d66d8e04ecdafd7a30c50aa1c9a (table rebalance_test_table_13 [id=4d62f84fd7bc47e8a1f2de384d64b787]) successfully deleted
I20240503 13:56:43.578157 12911 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=d8f2a259ec1043d99817c72d96665386] alter complete (version 2)
I20240503 13:56:43.607496 15623 consensus_queue.cc:237] T 6bca3e19362842fe928adb84e7c9297a P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:43.608892 15758 raft_consensus.cc:1261] T 6bca3e19362842fe928adb84e7c9297a P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Refusing update from remote peer 009f849ee7b54c80a3f5264695047e4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:43.609294 19586 consensus_queue.cc:1035] T 6bca3e19362842fe928adb84e7c9297a P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:56:43.609671 15889 raft_consensus.cc:1261] T 6bca3e19362842fe928adb84e7c9297a P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Refusing update from remote peer 009f849ee7b54c80a3f5264695047e4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:43.610096 19586 consensus_queue.cc:1035] T 6bca3e19362842fe928adb84e7c9297a P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 13:56:43.611790 15558 consensus_peers.cc:485] T 6bca3e19362842fe928adb84e7c9297a P 009f849ee7b54c80a3f5264695047e4d -> Peer 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109): Couldn't send request to peer 7690ee79577c45e3a2c8f8b552c0bff8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6bca3e19362842fe928adb84e7c9297a. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:43.612934 19841 raft_consensus.cc:482] T 5eeac82f64ed4e458ed86e452d70430d P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:43.613027 19841 raft_consensus.cc:504] T 5eeac82f64ed4e458ed86e452d70430d P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:43.613384 19841 leader_election.cc:288] T 5eeac82f64ed4e458ed86e452d70430d P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387), 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951)
I20240503 13:56:43.613538 19586 raft_consensus.cc:2932] T 6bca3e19362842fe928adb84e7c9297a P 009f849ee7b54c80a3f5264695047e4d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } }
I20240503 13:56:43.614686 13183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5eeac82f64ed4e458ed86e452d70430d" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770" is_pre_election: true
I20240503 13:56:43.614805 13183 raft_consensus.cc:2454] T 5eeac82f64ed4e458ed86e452d70430d P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 0.
I20240503 13:56:43.615126 15757 raft_consensus.cc:2932] T 6bca3e19362842fe928adb84e7c9297a P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } }
I20240503 13:56:43.615331 15623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5eeac82f64ed4e458ed86e452d70430d" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d" is_pre_election: true
I20240503 13:56:43.615432 15623 raft_consensus.cc:2454] T 5eeac82f64ed4e458ed86e452d70430d P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 0.
I20240503 13:56:43.615705 15691 leader_election.cc:302] T 5eeac82f64ed4e458ed86e452d70430d P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 76c386bef61a4bd5a907266a5118ffe2; no voters:
I20240503 13:56:43.615706 15888 raft_consensus.cc:2932] T 6bca3e19362842fe928adb84e7c9297a P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } }
I20240503 13:56:43.616044 19841 raft_consensus.cc:2781] T 5eeac82f64ed4e458ed86e452d70430d P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:43.616096 19841 raft_consensus.cc:482] T 5eeac82f64ed4e458ed86e452d70430d P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:43.616130 19841 raft_consensus.cc:3037] T 5eeac82f64ed4e458ed86e452d70430d P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:43.617130 19841 raft_consensus.cc:504] T 5eeac82f64ed4e458ed86e452d70430d P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:43.617468 19841 leader_election.cc:288] T 5eeac82f64ed4e458ed86e452d70430d P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 election: Requested vote from peers 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387), 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951)
I20240503 13:56:43.617903 15623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5eeac82f64ed4e458ed86e452d70430d" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d"
I20240503 13:56:43.618000 15623 raft_consensus.cc:3037] T 5eeac82f64ed4e458ed86e452d70430d P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:43.617910 12903 catalog_manager.cc:5484] T 6bca3e19362842fe928adb84e7c9297a P 009f849ee7b54c80a3f5264695047e4d reported cstate change: config changed from index -1 to 2, VOTER 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68) added. New cstate: current_term: 1 leader_uuid: "009f849ee7b54c80a3f5264695047e4d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } }
I20240503 13:56:43.619009 15623 raft_consensus.cc:2454] T 5eeac82f64ed4e458ed86e452d70430d P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 1.
I20240503 13:56:43.619479 15691 leader_election.cc:302] T 5eeac82f64ed4e458ed86e452d70430d P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 76c386bef61a4bd5a907266a5118ffe2; no voters:
I20240503 13:56:43.619849 19618 raft_consensus.cc:2781] T 5eeac82f64ed4e458ed86e452d70430d P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:43.619918 19618 raft_consensus.cc:686] T 5eeac82f64ed4e458ed86e452d70430d P 76c386bef61a4bd5a907266a5118ffe2 [term 1 LEADER]: Becoming Leader. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Running, Role: LEADER
I20240503 13:56:43.620699 13183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5eeac82f64ed4e458ed86e452d70430d" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770"
I20240503 13:56:43.620807 13183 raft_consensus.cc:3037] T 5eeac82f64ed4e458ed86e452d70430d P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:43.621177 19618 consensus_queue.cc:237] T 5eeac82f64ed4e458ed86e452d70430d P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:43.621793 13183 raft_consensus.cc:2454] T 5eeac82f64ed4e458ed86e452d70430d P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 1.
I20240503 13:56:43.626277 12911 catalog_manager.cc:5484] T 5eeac82f64ed4e458ed86e452d70430d P 76c386bef61a4bd5a907266a5118ffe2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71). New cstate: current_term: 1 leader_uuid: "76c386bef61a4bd5a907266a5118ffe2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:43.645318 13052 consensus_queue.cc:237] T 023517b9d4ee43f497d98c4d7a6037df P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:43.646595 13183 raft_consensus.cc:1261] T 023517b9d4ee43f497d98c4d7a6037df P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Refusing update from remote peer 60ba3ffe3b7746469834cd91c84be14e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:56:43.646984 20870 consensus_queue.cc:1035] T 023517b9d4ee43f497d98c4d7a6037df P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 13:56:43.647145 12987 consensus_peers.cc:485] T 023517b9d4ee43f497d98c4d7a6037df P 60ba3ffe3b7746469834cd91c84be14e -> Peer 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951): Couldn't send request to peer 009f849ee7b54c80a3f5264695047e4d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 023517b9d4ee43f497d98c4d7a6037df. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:43.647178 13316 raft_consensus.cc:1261] T 023517b9d4ee43f497d98c4d7a6037df P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Refusing update from remote peer 60ba3ffe3b7746469834cd91c84be14e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:56:43.647501 20870 consensus_queue.cc:1035] T 023517b9d4ee43f497d98c4d7a6037df P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:43.649466 20870 raft_consensus.cc:2932] T 023517b9d4ee43f497d98c4d7a6037df P 60ba3ffe3b7746469834cd91c84be14e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 009f849ee7b54c80a3f5264695047e4d (127.6.98.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:43.649880 13316 raft_consensus.cc:2932] T 023517b9d4ee43f497d98c4d7a6037df P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 009f849ee7b54c80a3f5264695047e4d (127.6.98.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:43.652069 13183 raft_consensus.cc:2932] T 023517b9d4ee43f497d98c4d7a6037df P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 009f849ee7b54c80a3f5264695047e4d (127.6.98.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:43.652624 12911 catalog_manager.cc:5484] T 023517b9d4ee43f497d98c4d7a6037df P ba0194098aa84449ae17e8a0d4c43703 reported cstate change: config changed from index -1 to 4, VOTER 009f849ee7b54c80a3f5264695047e4d (127.6.98.70) added. New cstate: current_term: 1 leader_uuid: "60ba3ffe3b7746469834cd91c84be14e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:43.672612 13316 consensus_queue.cc:237] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:43.673964 13445 raft_consensus.cc:1261] T 397f0e3e877d4acd9157bdf99bee6255 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:56:43.674000 13183 raft_consensus.cc:1261] T 397f0e3e877d4acd9157bdf99bee6255 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 13:56:43.674393 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 397f0e3e877d4acd9157bdf99bee6255. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:43.674526 20215 consensus_queue.cc:1035] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:43.674732 18914 consensus_queue.cc:1035] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:43.676898 18914 raft_consensus.cc:2932] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:43.677328 13445 raft_consensus.cc:2932] T 397f0e3e877d4acd9157bdf99bee6255 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:43.678910 13183 raft_consensus.cc:2932] T 397f0e3e877d4acd9157bdf99bee6255 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:43.681006 12904 catalog_manager.cc:5484] T 397f0e3e877d4acd9157bdf99bee6255 P 7690ee79577c45e3a2c8f8b552c0bff8 reported cstate change: config changed from index -1 to 4, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) added. New cstate: current_term: 1 leader_uuid: "ba0194098aa84449ae17e8a0d4c43703" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:43.682161 12906 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:56:43.688901 20629 tablet.cc:1722] T b537b5d76f234a5bb58ee4a6b28108ad P 61b887f0d40d497badeca5604c1f0546: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:43.689493 21062 raft_consensus.cc:482] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:43.689575 21062 raft_consensus.cc:504] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:43.689888 21062 leader_election.cc:288] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:43.690703 15623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2df7186d52a46fea36c672a22a1a0f0" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d" is_pre_election: true
I20240503 13:56:43.690832 15623 raft_consensus.cc:2454] T c2df7186d52a46fea36c672a22a1a0f0 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 0.
I20240503 13:56:43.690927 15888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2df7186d52a46fea36c672a22a1a0f0" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7" is_pre_election: true
I20240503 13:56:43.691087 15888 raft_consensus.cc:2454] T c2df7186d52a46fea36c672a22a1a0f0 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 0.
I20240503 13:56:43.691164 15426 leader_election.cc:302] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 61b887f0d40d497badeca5604c1f0546; no voters:
I20240503 13:56:43.691308 20626 tablet.cc:1722] T b537b5d76f234a5bb58ee4a6b28108ad P 009f849ee7b54c80a3f5264695047e4d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:43.692277 20722 raft_consensus.cc:2781] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:43.692340 20722 raft_consensus.cc:482] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:43.692374 20722 raft_consensus.cc:3037] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:43.693318 20722 raft_consensus.cc:504] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:43.693950 20722 leader_election.cc:288] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 election: Requested vote from peers 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:43.694370 21132 tablet.cc:1722] T b537b5d76f234a5bb58ee4a6b28108ad P 60ba3ffe3b7746469834cd91c84be14e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:43.696767 12906 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table_new_name [id=d8f2a259ec1043d99817c72d96665386] alter complete (version 3)
I20240503 13:56:43.697602 15624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2df7186d52a46fea36c672a22a1a0f0" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d"
I20240503 13:56:43.697737 15624 raft_consensus.cc:3037] T c2df7186d52a46fea36c672a22a1a0f0 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:43.698742 15624 raft_consensus.cc:2454] T c2df7186d52a46fea36c672a22a1a0f0 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 1.
I20240503 13:56:43.699146 15888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2df7186d52a46fea36c672a22a1a0f0" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7"
I20240503 13:56:43.699187 15426 leader_election.cc:302] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 61b887f0d40d497badeca5604c1f0546; no voters:
I20240503 13:56:43.699265 15888 raft_consensus.cc:3037] T c2df7186d52a46fea36c672a22a1a0f0 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:43.699926 21062 raft_consensus.cc:2781] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:43.700021 21062 raft_consensus.cc:686] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 [term 1 LEADER]: Becoming Leader. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Running, Role: LEADER
I20240503 13:56:43.700155 21062 consensus_queue.cc:237] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:43.700222 15888 raft_consensus.cc:2454] T c2df7186d52a46fea36c672a22a1a0f0 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 1.
I20240503 13:56:43.702419 12903 catalog_manager.cc:5484] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61b887f0d40d497badeca5604c1f0546 (127.6.98.69). New cstate: current_term: 1 leader_uuid: "61b887f0d40d497badeca5604c1f0546" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:43.713979 15757 consensus_queue.cc:237] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:43.715520 13052 raft_consensus.cc:1261] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Refusing update from remote peer 76c386bef61a4bd5a907266a5118ffe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 13:56:43.715523 15694 consensus_peers.cc:485] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279): Couldn't send request to peer ba0194098aa84449ae17e8a0d4c43703. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fcb0aa9fce544ae2a6f1ca96c5315c36. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:43.715569 15624 raft_consensus.cc:1261] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Refusing update from remote peer 76c386bef61a4bd5a907266a5118ffe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:43.715893 19841 consensus_queue.cc:1035] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:56:43.716073 19618 consensus_queue.cc:1035] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:56:43.718056 19841 raft_consensus.cc:2932] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:43.718456 15624 raft_consensus.cc:2932] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:43.720330 13052 raft_consensus.cc:2932] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:43.723256 12906 catalog_manager.cc:5484] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 76c386bef61a4bd5a907266a5118ffe2 reported cstate change: config changed from index -1 to 2, VOTER ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) added. New cstate: current_term: 1 leader_uuid: "76c386bef61a4bd5a907266a5118ffe2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:43.752656 13183 consensus_queue.cc:237] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:43.754110 15758 raft_consensus.cc:1261] T cbd6d38ea18b484d80f4a918148efd3a P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Refusing update from remote peer 8ccf5a4ac450414c9d3579bd5800b770: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 13:56:43.754153 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cbd6d38ea18b484d80f4a918148efd3a. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:43.754176 15493 raft_consensus.cc:1261] T cbd6d38ea18b484d80f4a918148efd3a P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Refusing update from remote peer 8ccf5a4ac450414c9d3579bd5800b770: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:43.754570 21272 consensus_queue.cc:1035] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:56:43.754753 19019 consensus_queue.cc:1035] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:56:43.757175 19019 raft_consensus.cc:2932] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:43.757571 15493 raft_consensus.cc:2932] T cbd6d38ea18b484d80f4a918148efd3a P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:43.759042 15758 raft_consensus.cc:2932] T cbd6d38ea18b484d80f4a918148efd3a P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:43.760556 12903 catalog_manager.cc:5484] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 reported cstate change: config changed from index -1 to 2, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) added. New cstate: current_term: 1 leader_uuid: "8ccf5a4ac450414c9d3579bd5800b770" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:43.781991 13445 consensus_queue.cc:237] T 356ccd6fdf25470dba2a79367a37128e P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:43.783457 15758 raft_consensus.cc:1261] T 356ccd6fdf25470dba2a79367a37128e P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Refusing update from remote peer 7690ee79577c45e3a2c8f8b552c0bff8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:43.783488 15493 raft_consensus.cc:1261] T 356ccd6fdf25470dba2a79367a37128e P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Refusing update from remote peer 7690ee79577c45e3a2c8f8b552c0bff8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 13:56:43.783521 13383 consensus_peers.cc:485] T 356ccd6fdf25470dba2a79367a37128e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387): Couldn't send request to peer 8ccf5a4ac450414c9d3579bd5800b770. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 356ccd6fdf25470dba2a79367a37128e. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:43.783847 21292 consensus_queue.cc:1035] T 356ccd6fdf25470dba2a79367a37128e P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:56:43.784116 20120 consensus_queue.cc:1035] T 356ccd6fdf25470dba2a79367a37128e P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:56:43.785843 21292 raft_consensus.cc:2932] T 356ccd6fdf25470dba2a79367a37128e P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } }
I20240503 13:56:43.786257 15493 raft_consensus.cc:2932] T 356ccd6fdf25470dba2a79367a37128e P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } }
I20240503 13:56:43.788254 15758 raft_consensus.cc:2932] T 356ccd6fdf25470dba2a79367a37128e P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } }
I20240503 13:56:43.788913 12911 catalog_manager.cc:5484] T 356ccd6fdf25470dba2a79367a37128e P 61b887f0d40d497badeca5604c1f0546 reported cstate change: config changed from index -1 to 2, VOTER 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66) added. New cstate: current_term: 1 leader_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } }
I20240503 13:56:43.801066 12906 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:43.801218 12906 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:43.803659 12906 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Sending DeleteTablet for 3 replicas of tablet b537b5d76f234a5bb58ee4a6b28108ad
I20240503 13:56:43.804224 13032 tablet_service.cc:1514] Processing DeleteTablet for tablet b537b5d76f234a5bb58ee4a6b28108ad with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:43 UTC) from {username='slave'} at 127.0.0.1:59458
I20240503 13:56:43.804260 15605 tablet_service.cc:1514] Processing DeleteTablet for tablet b537b5d76f234a5bb58ee4a6b28108ad with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:43 UTC) from {username='slave'} at 127.0.0.1:60718
I20240503 13:56:43.804642 21307 tablet_replica.cc:330] T b537b5d76f234a5bb58ee4a6b28108ad P 60ba3ffe3b7746469834cd91c84be14e: stopping tablet replica
I20240503 13:56:43.804651 21308 tablet_replica.cc:330] T b537b5d76f234a5bb58ee4a6b28108ad P 009f849ee7b54c80a3f5264695047e4d: stopping tablet replica
I20240503 13:56:43.804733 15471 tablet_service.cc:1514] Processing DeleteTablet for tablet b537b5d76f234a5bb58ee4a6b28108ad with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:43 UTC) from {username='slave'} at 127.0.0.1:58758
I20240503 13:56:43.804822 21307 raft_consensus.cc:2229] T b537b5d76f234a5bb58ee4a6b28108ad P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:43.804822 21308 raft_consensus.cc:2229] T b537b5d76f234a5bb58ee4a6b28108ad P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:43.804941 21307 raft_consensus.cc:2258] T b537b5d76f234a5bb58ee4a6b28108ad P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:43.804941 21308 raft_consensus.cc:2258] T b537b5d76f234a5bb58ee4a6b28108ad P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:43.805526 21307 ts_tablet_manager.cc:1875] T b537b5d76f234a5bb58ee4a6b28108ad P 60ba3ffe3b7746469834cd91c84be14e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:43.805538 21308 ts_tablet_manager.cc:1875] T b537b5d76f234a5bb58ee4a6b28108ad P 009f849ee7b54c80a3f5264695047e4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:43.806543 21309 tablet_replica.cc:330] T b537b5d76f234a5bb58ee4a6b28108ad P 61b887f0d40d497badeca5604c1f0546: stopping tablet replica
I20240503 13:56:43.806713 21309 raft_consensus.cc:2229] T b537b5d76f234a5bb58ee4a6b28108ad P 61b887f0d40d497badeca5604c1f0546 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:56:43.806931 21309 raft_consensus.cc:2258] T b537b5d76f234a5bb58ee4a6b28108ad P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:43.807512 21309 ts_tablet_manager.cc:1875] T b537b5d76f234a5bb58ee4a6b28108ad P 61b887f0d40d497badeca5604c1f0546: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:43.808276 21307 ts_tablet_manager.cc:1888] T b537b5d76f234a5bb58ee4a6b28108ad P 60ba3ffe3b7746469834cd91c84be14e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:43.808393 21307 log.cc:1196] T b537b5d76f234a5bb58ee4a6b28108ad P 60ba3ffe3b7746469834cd91c84be14e: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal/wals/b537b5d76f234a5bb58ee4a6b28108ad
I20240503 13:56:43.808756 21307 ts_tablet_manager.cc:1909] T b537b5d76f234a5bb58ee4a6b28108ad P 60ba3ffe3b7746469834cd91c84be14e: Deleting consensus metadata
I20240503 13:56:43.810997 21309 ts_tablet_manager.cc:1888] T b537b5d76f234a5bb58ee4a6b28108ad P 61b887f0d40d497badeca5604c1f0546: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:43.811072 21309 log.cc:1196] T b537b5d76f234a5bb58ee4a6b28108ad P 61b887f0d40d497badeca5604c1f0546: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/wal/wals/b537b5d76f234a5bb58ee4a6b28108ad
I20240503 13:56:43.811362 21309 ts_tablet_manager.cc:1909] T b537b5d76f234a5bb58ee4a6b28108ad P 61b887f0d40d497badeca5604c1f0546: Deleting consensus metadata
I20240503 13:56:43.811975 12899 catalog_manager.cc:4830] TS 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153): tablet b537b5d76f234a5bb58ee4a6b28108ad (table rebalancer_dynamic_table_new_name [id=d8f2a259ec1043d99817c72d96665386]) successfully deleted
I20240503 13:56:43.812714 12897 catalog_manager.cc:4830] TS 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695): tablet b537b5d76f234a5bb58ee4a6b28108ad (table rebalancer_dynamic_table_new_name [id=d8f2a259ec1043d99817c72d96665386]) successfully deleted
I20240503 13:56:43.814433 21308 ts_tablet_manager.cc:1888] T b537b5d76f234a5bb58ee4a6b28108ad P 009f849ee7b54c80a3f5264695047e4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:43.814615 21308 log.cc:1196] T b537b5d76f234a5bb58ee4a6b28108ad P 009f849ee7b54c80a3f5264695047e4d: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/wal/wals/b537b5d76f234a5bb58ee4a6b28108ad
I20240503 13:56:43.815225 21308 ts_tablet_manager.cc:1909] T b537b5d76f234a5bb58ee4a6b28108ad P 009f849ee7b54c80a3f5264695047e4d: Deleting consensus metadata
I20240503 13:56:43.816205 12896 catalog_manager.cc:4830] TS 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951): tablet b537b5d76f234a5bb58ee4a6b28108ad (table rebalancer_dynamic_table_new_name [id=d8f2a259ec1043d99817c72d96665386]) successfully deleted
I20240503 13:56:43.891351 15471 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:56:43.891674 15605 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 13:56:43.892596 13032 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:43.900509 15736 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 13:56:43.905346 13424 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:43.907083 13164 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 13:56:43.907135 13296 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 13:56:43.908704 12905 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:56:43.916401 13164 tablet_service.cc:1467] Processing CreateTablet for tablet 0040894be67a49f5ab84eb0dc1f309a8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ce0ba30e09fd4ea789864f5d1b8d2a1e]), partition=
I20240503 13:56:43.916694 13164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0040894be67a49f5ab84eb0dc1f309a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:43.916711 13296 tablet_service.cc:1467] Processing CreateTablet for tablet 0040894be67a49f5ab84eb0dc1f309a8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ce0ba30e09fd4ea789864f5d1b8d2a1e]), partition=
I20240503 13:56:43.916929 13296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0040894be67a49f5ab84eb0dc1f309a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:43.919103 21130 tablet_bootstrap.cc:492] T 0040894be67a49f5ab84eb0dc1f309a8 P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:43.919972 21130 tablet_bootstrap.cc:654] T 0040894be67a49f5ab84eb0dc1f309a8 P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:43.921247 15605 tablet_service.cc:1467] Processing CreateTablet for tablet 0040894be67a49f5ab84eb0dc1f309a8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ce0ba30e09fd4ea789864f5d1b8d2a1e]), partition=
I20240503 13:56:43.921277 21351 tablet_bootstrap.cc:492] T 0040894be67a49f5ab84eb0dc1f309a8 P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:43.921468 15605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0040894be67a49f5ab84eb0dc1f309a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:43.922122 21351 tablet_bootstrap.cc:654] T 0040894be67a49f5ab84eb0dc1f309a8 P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:43.923280 20796 tablet_bootstrap.cc:492] T 0040894be67a49f5ab84eb0dc1f309a8 P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:43.923416 21351 tablet_bootstrap.cc:492] T 0040894be67a49f5ab84eb0dc1f309a8 P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:43.923430 21130 tablet_bootstrap.cc:492] T 0040894be67a49f5ab84eb0dc1f309a8 P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:43.923506 21130 ts_tablet_manager.cc:1367] T 0040894be67a49f5ab84eb0dc1f309a8 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 13:56:43.923626 21351 ts_tablet_manager.cc:1367] T 0040894be67a49f5ab84eb0dc1f309a8 P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:56:43.923966 21130 raft_consensus.cc:348] T 0040894be67a49f5ab84eb0dc1f309a8 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:43.924080 21130 raft_consensus.cc:374] T 0040894be67a49f5ab84eb0dc1f309a8 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:43.924113 21130 raft_consensus.cc:729] T 0040894be67a49f5ab84eb0dc1f309a8 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:43.924059 21351 raft_consensus.cc:348] T 0040894be67a49f5ab84eb0dc1f309a8 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:43.924170 20796 tablet_bootstrap.cc:654] T 0040894be67a49f5ab84eb0dc1f309a8 P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:43.924198 21351 raft_consensus.cc:374] T 0040894be67a49f5ab84eb0dc1f309a8 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:43.924268 21351 raft_consensus.cc:729] T 0040894be67a49f5ab84eb0dc1f309a8 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:43.924244 21130 consensus_queue.cc:260] T 0040894be67a49f5ab84eb0dc1f309a8 P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:43.924376 21351 consensus_queue.cc:260] T 0040894be67a49f5ab84eb0dc1f309a8 P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:43.924562 21130 ts_tablet_manager.cc:1398] T 0040894be67a49f5ab84eb0dc1f309a8 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:43.926226 21351 ts_tablet_manager.cc:1398] T 0040894be67a49f5ab84eb0dc1f309a8 P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:56:43.931202 20722 consensus_queue.cc:1035] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:43.949952 20796 tablet_bootstrap.cc:492] T 0040894be67a49f5ab84eb0dc1f309a8 P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:43.950083 20796 ts_tablet_manager.cc:1367] T 0040894be67a49f5ab84eb0dc1f309a8 P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.000s
I20240503 13:56:43.950542 20796 raft_consensus.cc:348] T 0040894be67a49f5ab84eb0dc1f309a8 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:43.950655 20796 raft_consensus.cc:374] T 0040894be67a49f5ab84eb0dc1f309a8 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:43.950682 20796 raft_consensus.cc:729] T 0040894be67a49f5ab84eb0dc1f309a8 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:43.950790 20796 consensus_queue.cc:260] T 0040894be67a49f5ab84eb0dc1f309a8 P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:43.951051 20796 ts_tablet_manager.cc:1398] T 0040894be67a49f5ab84eb0dc1f309a8 P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:43.951275 15868 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:56:43.957275 20346 consensus_queue.cc:1035] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:43.997961 21255 consensus_queue.cc:1035] T 5eeac82f64ed4e458ed86e452d70430d P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:44.001446 21252 consensus_queue.cc:1035] T 5eeac82f64ed4e458ed86e452d70430d P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:44.063751 21401 ts_tablet_manager.cc:902] T 023517b9d4ee43f497d98c4d7a6037df P 009f849ee7b54c80a3f5264695047e4d: Initiating tablet copy from peer 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:44.065253 21401 tablet_copy_client.cc:308] T 023517b9d4ee43f497d98c4d7a6037df P 009f849ee7b54c80a3f5264695047e4d: tablet copy: Beginning tablet copy session from remote peer at address 127.6.98.65:34695
I20240503 13:56:44.065810 13074 tablet_copy_service.cc:140] P 60ba3ffe3b7746469834cd91c84be14e: Received BeginTabletCopySession request for tablet 023517b9d4ee43f497d98c4d7a6037df from peer 009f849ee7b54c80a3f5264695047e4d ({username='slave'} at 127.6.98.70:47015)
I20240503 13:56:44.066207 13074 tablet_copy_service.cc:161] P 60ba3ffe3b7746469834cd91c84be14e: Beginning new tablet copy session on tablet 023517b9d4ee43f497d98c4d7a6037df from peer 009f849ee7b54c80a3f5264695047e4d at {username='slave'} at 127.6.98.70:47015: session id = 009f849ee7b54c80a3f5264695047e4d-023517b9d4ee43f497d98c4d7a6037df
I20240503 13:56:44.069445 20818 raft_consensus.cc:482] T 0040894be67a49f5ab84eb0dc1f309a8 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:44.069854 20818 raft_consensus.cc:504] T 0040894be67a49f5ab84eb0dc1f309a8 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:44.070606 20818 leader_election.cc:288] T 0040894be67a49f5ab84eb0dc1f309a8 P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279), 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951)
I20240503 13:56:44.071043 13316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0040894be67a49f5ab84eb0dc1f309a8" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703" is_pre_election: true
I20240503 13:56:44.071034 15624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0040894be67a49f5ab84eb0dc1f309a8" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d" is_pre_election: true
I20240503 13:56:44.071170 15624 raft_consensus.cc:2454] T 0040894be67a49f5ab84eb0dc1f309a8 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 0.
I20240503 13:56:44.071170 13316 raft_consensus.cc:2454] T 0040894be67a49f5ab84eb0dc1f309a8 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 0.
I20240503 13:56:44.071467 13121 leader_election.cc:302] T 0040894be67a49f5ab84eb0dc1f309a8 P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ccf5a4ac450414c9d3579bd5800b770, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:44.071854 21275 raft_consensus.cc:2781] T 0040894be67a49f5ab84eb0dc1f309a8 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:44.071910 21275 raft_consensus.cc:482] T 0040894be67a49f5ab84eb0dc1f309a8 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:44.071954 21275 raft_consensus.cc:3037] T 0040894be67a49f5ab84eb0dc1f309a8 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:44.074683 21275 raft_consensus.cc:504] T 0040894be67a49f5ab84eb0dc1f309a8 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:44.075065 21275 leader_election.cc:288] T 0040894be67a49f5ab84eb0dc1f309a8 P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 election: Requested vote from peers ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279), 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951)
I20240503 13:56:44.075384 15624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0040894be67a49f5ab84eb0dc1f309a8" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d"
I20240503 13:56:44.075443 13316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0040894be67a49f5ab84eb0dc1f309a8" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703"
I20240503 13:56:44.075528 15624 raft_consensus.cc:3037] T 0040894be67a49f5ab84eb0dc1f309a8 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:44.075538 13316 raft_consensus.cc:3037] T 0040894be67a49f5ab84eb0dc1f309a8 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:44.076506 13316 raft_consensus.cc:2454] T 0040894be67a49f5ab84eb0dc1f309a8 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 1.
I20240503 13:56:44.076527 15624 raft_consensus.cc:2454] T 0040894be67a49f5ab84eb0dc1f309a8 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 1.
I20240503 13:56:44.076956 13118 leader_election.cc:302] T 0040894be67a49f5ab84eb0dc1f309a8 P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 8ccf5a4ac450414c9d3579bd5800b770; no voters:
I20240503 13:56:44.077126 21275 raft_consensus.cc:2781] T 0040894be67a49f5ab84eb0dc1f309a8 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:44.077224 21275 raft_consensus.cc:686] T 0040894be67a49f5ab84eb0dc1f309a8 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 LEADER]: Becoming Leader. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Running, Role: LEADER
I20240503 13:56:44.077356 21275 consensus_queue.cc:237] T 0040894be67a49f5ab84eb0dc1f309a8 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:44.076646 13074 tablet_copy_source_session.cc:205] T 023517b9d4ee43f497d98c4d7a6037df P 60ba3ffe3b7746469834cd91c84be14e: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:56:44.079643 12912 catalog_manager.cc:5484] T 0040894be67a49f5ab84eb0dc1f309a8 P 8ccf5a4ac450414c9d3579bd5800b770 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66). New cstate: current_term: 1 leader_uuid: "8ccf5a4ac450414c9d3579bd5800b770" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:44.081012 21401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 023517b9d4ee43f497d98c4d7a6037df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:44.081924 21407 ts_tablet_manager.cc:902] T 6bca3e19362842fe928adb84e7c9297a P 7690ee79577c45e3a2c8f8b552c0bff8: Initiating tablet copy from peer 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951)
I20240503 13:56:44.082551 21407 tablet_copy_client.cc:308] T 6bca3e19362842fe928adb84e7c9297a P 7690ee79577c45e3a2c8f8b552c0bff8: tablet copy: Beginning tablet copy session from remote peer at address 127.6.98.70:34951
I20240503 13:56:44.083101 15645 tablet_copy_service.cc:140] P 009f849ee7b54c80a3f5264695047e4d: Received BeginTabletCopySession request for tablet 6bca3e19362842fe928adb84e7c9297a from peer 7690ee79577c45e3a2c8f8b552c0bff8 ({username='slave'} at 127.6.98.68:58341)
I20240503 13:56:44.083195 15645 tablet_copy_service.cc:161] P 009f849ee7b54c80a3f5264695047e4d: Beginning new tablet copy session on tablet 6bca3e19362842fe928adb84e7c9297a from peer 7690ee79577c45e3a2c8f8b552c0bff8 at {username='slave'} at 127.6.98.68:58341: session id = 7690ee79577c45e3a2c8f8b552c0bff8-6bca3e19362842fe928adb84e7c9297a
I20240503 13:56:44.084134 15645 tablet_copy_source_session.cc:205] T 6bca3e19362842fe928adb84e7c9297a P 009f849ee7b54c80a3f5264695047e4d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:56:44.084280 21401 tablet_copy_client.cc:783] T 023517b9d4ee43f497d98c4d7a6037df P 009f849ee7b54c80a3f5264695047e4d: tablet copy: Starting download of 0 data blocks...
I20240503 13:56:44.084426 21401 tablet_copy_client.cc:647] T 023517b9d4ee43f497d98c4d7a6037df P 009f849ee7b54c80a3f5264695047e4d: tablet copy: Starting download of 1 WAL segments...
I20240503 13:56:44.085947 21401 tablet_copy_client.cc:523] T 023517b9d4ee43f497d98c4d7a6037df P 009f849ee7b54c80a3f5264695047e4d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:56:44.086125 21407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bca3e19362842fe928adb84e7c9297a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:44.087103 21401 tablet_bootstrap.cc:492] T 023517b9d4ee43f497d98c4d7a6037df P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:44.088305 21407 tablet_copy_client.cc:783] T 6bca3e19362842fe928adb84e7c9297a P 7690ee79577c45e3a2c8f8b552c0bff8: tablet copy: Starting download of 0 data blocks...
I20240503 13:56:44.088443 21407 tablet_copy_client.cc:647] T 6bca3e19362842fe928adb84e7c9297a P 7690ee79577c45e3a2c8f8b552c0bff8: tablet copy: Starting download of 1 WAL segments...
I20240503 13:56:44.091852 21407 tablet_copy_client.cc:523] T 6bca3e19362842fe928adb84e7c9297a P 7690ee79577c45e3a2c8f8b552c0bff8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:56:44.094832 21407 tablet_bootstrap.cc:492] T 6bca3e19362842fe928adb84e7c9297a P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:44.106158 15605 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 13:56:44.107733 15471 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:56:44.112660 13032 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:44.119832 21401 tablet_bootstrap.cc:492] T 023517b9d4ee43f497d98c4d7a6037df P 009f849ee7b54c80a3f5264695047e4d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:56:44.120189 21401 tablet_bootstrap.cc:492] T 023517b9d4ee43f497d98c4d7a6037df P 009f849ee7b54c80a3f5264695047e4d: Bootstrap complete.
I20240503 13:56:44.120334 21407 tablet_bootstrap.cc:492] T 6bca3e19362842fe928adb84e7c9297a P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:56:44.120625 21407 tablet_bootstrap.cc:492] T 6bca3e19362842fe928adb84e7c9297a P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap complete.
I20240503 13:56:44.120729 21401 ts_tablet_manager.cc:1367] T 023517b9d4ee43f497d98c4d7a6037df P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.034s user 0.026s sys 0.000s
I20240503 13:56:44.120790 21407 ts_tablet_manager.cc:1367] T 6bca3e19362842fe928adb84e7c9297a P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.000s
I20240503 13:56:44.121281 21407 raft_consensus.cc:348] T 6bca3e19362842fe928adb84e7c9297a P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:44.121412 21407 raft_consensus.cc:729] T 6bca3e19362842fe928adb84e7c9297a P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:44.121543 21407 consensus_queue.cc:260] T 6bca3e19362842fe928adb84e7c9297a P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:44.121625 21401 raft_consensus.cc:348] T 023517b9d4ee43f497d98c4d7a6037df P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:44.121831 21401 raft_consensus.cc:729] T 023517b9d4ee43f497d98c4d7a6037df P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:44.121868 21407 ts_tablet_manager.cc:1398] T 6bca3e19362842fe928adb84e7c9297a P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:44.122049 21401 consensus_queue.cc:260] T 023517b9d4ee43f497d98c4d7a6037df P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:44.122512 21401 ts_tablet_manager.cc:1398] T 023517b9d4ee43f497d98c4d7a6037df P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:44.122536 15645 tablet_copy_service.cc:342] P 009f849ee7b54c80a3f5264695047e4d: Request end of tablet copy session 7690ee79577c45e3a2c8f8b552c0bff8-6bca3e19362842fe928adb84e7c9297a received from {username='slave'} at 127.6.98.68:58341
I20240503 13:56:44.122913 15645 tablet_copy_service.cc:433] P 009f849ee7b54c80a3f5264695047e4d: ending tablet copy session 7690ee79577c45e3a2c8f8b552c0bff8-6bca3e19362842fe928adb84e7c9297a on tablet 6bca3e19362842fe928adb84e7c9297a with peer 7690ee79577c45e3a2c8f8b552c0bff8
I20240503 13:56:44.124156 13074 tablet_copy_service.cc:342] P 60ba3ffe3b7746469834cd91c84be14e: Request end of tablet copy session 009f849ee7b54c80a3f5264695047e4d-023517b9d4ee43f497d98c4d7a6037df received from {username='slave'} at 127.6.98.70:47015
I20240503 13:56:44.124258 13074 tablet_copy_service.cc:433] P 60ba3ffe3b7746469834cd91c84be14e: ending tablet copy session 009f849ee7b54c80a3f5264695047e4d-023517b9d4ee43f497d98c4d7a6037df on tablet 023517b9d4ee43f497d98c4d7a6037df with peer 009f849ee7b54c80a3f5264695047e4d
I20240503 13:56:44.124577 15868 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:56:44.124809 13424 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:44.125722 13296 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 13:56:44.131291 13164 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 13:56:44.141784 20346 consensus_queue.cc:1035] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:44.146169 15736 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 13:56:44.186993 20722 consensus_queue.cc:1035] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:44.189425 12908 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:56:44.195415 15624 raft_consensus.cc:1261] T 0040894be67a49f5ab84eb0dc1f309a8 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Refusing update from remote peer 8ccf5a4ac450414c9d3579bd5800b770: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:44.195434 13316 raft_consensus.cc:1261] T 0040894be67a49f5ab84eb0dc1f309a8 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Refusing update from remote peer 8ccf5a4ac450414c9d3579bd5800b770: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:44.196015 20818 consensus_queue.cc:1035] T 0040894be67a49f5ab84eb0dc1f309a8 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:44.197381 20818 consensus_queue.cc:1035] T 0040894be67a49f5ab84eb0dc1f309a8 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:44.201227 12912 catalog_manager.cc:5576] TS ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279) does not have the latest schema for tablet 0040894be67a49f5ab84eb0dc1f309a8 (table rebalancer_dynamic_table [id=ce0ba30e09fd4ea789864f5d1b8d2a1e]). Expected version 1 got 0
I20240503 13:56:44.202160 21435 tablet.cc:1722] T 0040894be67a49f5ab84eb0dc1f309a8 P 8ccf5a4ac450414c9d3579bd5800b770: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:44.202754 21435 tablet.cc:1620] T 0040894be67a49f5ab84eb0dc1f309a8 P 8ccf5a4ac450414c9d3579bd5800b770: MemRowSet was empty: no flush needed.
I20240503 13:56:44.202922 12908 catalog_manager.cc:5576] TS 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951) does not have the latest schema for tablet 0040894be67a49f5ab84eb0dc1f309a8 (table rebalancer_dynamic_table [id=ce0ba30e09fd4ea789864f5d1b8d2a1e]). Expected version 1 got 0
I20240503 13:56:44.204273 21436 tablet.cc:1722] T 0040894be67a49f5ab84eb0dc1f309a8 P 009f849ee7b54c80a3f5264695047e4d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:44.205071 21436 tablet.cc:1620] T 0040894be67a49f5ab84eb0dc1f309a8 P 009f849ee7b54c80a3f5264695047e4d: MemRowSet was empty: no flush needed.
I20240503 13:56:44.205947 12908 catalog_manager.cc:5576] TS 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951) does not have the latest schema for tablet 0040894be67a49f5ab84eb0dc1f309a8 (table rebalancer_dynamic_table [id=ce0ba30e09fd4ea789864f5d1b8d2a1e]). Expected version 1 got 0
I20240503 13:56:44.208304 12909 catalog_manager.cc:5576] TS ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279) does not have the latest schema for tablet 0040894be67a49f5ab84eb0dc1f309a8 (table rebalancer_dynamic_table [id=ce0ba30e09fd4ea789864f5d1b8d2a1e]). Expected version 1 got 0
I20240503 13:56:44.206867 21441 tablet.cc:1722] T 0040894be67a49f5ab84eb0dc1f309a8 P ba0194098aa84449ae17e8a0d4c43703: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:44.208855 21441 tablet.cc:1620] T 0040894be67a49f5ab84eb0dc1f309a8 P ba0194098aa84449ae17e8a0d4c43703: MemRowSet was empty: no flush needed.
I20240503 13:56:44.212549 12909 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_extra_table_40"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:56:44.213132 12908 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=ce0ba30e09fd4ea789864f5d1b8d2a1e] alter complete (version 1)
I20240503 13:56:44.223659 13032 tablet_service.cc:1467] Processing CreateTablet for tablet fd61c523ce9241ff8c2a649e037bbfc4 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=76699e60112c467eb261bc198c3b8cb5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:44.223659 15605 tablet_service.cc:1467] Processing CreateTablet for tablet fd61c523ce9241ff8c2a649e037bbfc4 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=76699e60112c467eb261bc198c3b8cb5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:44.224015 13032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd61c523ce9241ff8c2a649e037bbfc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:44.224042 15605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd61c523ce9241ff8c2a649e037bbfc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:44.226223 20796 tablet_bootstrap.cc:492] T fd61c523ce9241ff8c2a649e037bbfc4 P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:44.226855 21444 tablet_bootstrap.cc:492] T fd61c523ce9241ff8c2a649e037bbfc4 P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:44.227118 20796 tablet_bootstrap.cc:654] T fd61c523ce9241ff8c2a649e037bbfc4 P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:44.227905 21444 tablet_bootstrap.cc:654] T fd61c523ce9241ff8c2a649e037bbfc4 P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:44.229002 21444 tablet_bootstrap.cc:492] T fd61c523ce9241ff8c2a649e037bbfc4 P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:44.229087 21444 ts_tablet_manager.cc:1367] T fd61c523ce9241ff8c2a649e037bbfc4 P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:56:44.229472 15472 tablet_service.cc:1467] Processing CreateTablet for tablet 29bae7c7b64244ce94e36360e740408d (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=76699e60112c467eb261bc198c3b8cb5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:44.229525 21444 raft_consensus.cc:348] T fd61c523ce9241ff8c2a649e037bbfc4 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:44.229619 21444 raft_consensus.cc:374] T fd61c523ce9241ff8c2a649e037bbfc4 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:44.229645 21444 raft_consensus.cc:729] T fd61c523ce9241ff8c2a649e037bbfc4 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:44.229743 15472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29bae7c7b64244ce94e36360e740408d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:44.229682 15737 tablet_service.cc:1467] Processing CreateTablet for tablet 29bae7c7b64244ce94e36360e740408d (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=76699e60112c467eb261bc198c3b8cb5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:44.229759 21444 consensus_queue.cc:260] T fd61c523ce9241ff8c2a649e037bbfc4 P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:44.229844 15471 tablet_service.cc:1467] Processing CreateTablet for tablet 54e6b2f0d7da4d58a55d8a369f3843ce (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=76699e60112c467eb261bc198c3b8cb5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:44.229935 15737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29bae7c7b64244ce94e36360e740408d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:44.230036 21444 ts_tablet_manager.cc:1398] T fd61c523ce9241ff8c2a649e037bbfc4 P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:44.230098 15471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54e6b2f0d7da4d58a55d8a369f3843ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:44.230466 15736 tablet_service.cc:1467] Processing CreateTablet for tablet fd61c523ce9241ff8c2a649e037bbfc4 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=76699e60112c467eb261bc198c3b8cb5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:44.230682 15736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd61c523ce9241ff8c2a649e037bbfc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:44.232072 15738 tablet_service.cc:1467] Processing CreateTablet for tablet 54e6b2f0d7da4d58a55d8a369f3843ce (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=76699e60112c467eb261bc198c3b8cb5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:44.232275 15738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54e6b2f0d7da4d58a55d8a369f3843ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:44.236047 13032 tablet_service.cc:1467] Processing CreateTablet for tablet 54e6b2f0d7da4d58a55d8a369f3843ce (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=76699e60112c467eb261bc198c3b8cb5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:44.236289 13032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54e6b2f0d7da4d58a55d8a369f3843ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:44.238377 21444 tablet_bootstrap.cc:492] T 54e6b2f0d7da4d58a55d8a369f3843ce P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:44.238659 21448 tablet_bootstrap.cc:492] T 29bae7c7b64244ce94e36360e740408d P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:44.239306 21444 tablet_bootstrap.cc:654] T 54e6b2f0d7da4d58a55d8a369f3843ce P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:44.239611 21448 tablet_bootstrap.cc:654] T 29bae7c7b64244ce94e36360e740408d P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:44.240396 21444 tablet_bootstrap.cc:492] T 54e6b2f0d7da4d58a55d8a369f3843ce P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:44.240478 21444 ts_tablet_manager.cc:1367] T 54e6b2f0d7da4d58a55d8a369f3843ce P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:56:44.240635 20796 tablet_bootstrap.cc:492] T fd61c523ce9241ff8c2a649e037bbfc4 P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:44.240713 20796 ts_tablet_manager.cc:1367] T fd61c523ce9241ff8c2a649e037bbfc4 P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240503 13:56:44.240998 21444 raft_consensus.cc:348] T 54e6b2f0d7da4d58a55d8a369f3843ce P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:44.241099 21444 raft_consensus.cc:374] T 54e6b2f0d7da4d58a55d8a369f3843ce P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:44.241130 21444 raft_consensus.cc:729] T 54e6b2f0d7da4d58a55d8a369f3843ce P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:44.241107 20796 raft_consensus.cc:348] T fd61c523ce9241ff8c2a649e037bbfc4 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:44.241194 20796 raft_consensus.cc:374] T fd61c523ce9241ff8c2a649e037bbfc4 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:44.241221 20796 raft_consensus.cc:729] T fd61c523ce9241ff8c2a649e037bbfc4 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:44.241250 21444 consensus_queue.cc:260] T 54e6b2f0d7da4d58a55d8a369f3843ce P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:44.241328 20796 consensus_queue.cc:260] T fd61c523ce9241ff8c2a649e037bbfc4 P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:44.241537 21444 ts_tablet_manager.cc:1398] T 54e6b2f0d7da4d58a55d8a369f3843ce P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:44.241600 20796 ts_tablet_manager.cc:1398] T fd61c523ce9241ff8c2a649e037bbfc4 P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:44.242223 15605 tablet_service.cc:1467] Processing CreateTablet for tablet 29bae7c7b64244ce94e36360e740408d (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=76699e60112c467eb261bc198c3b8cb5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:44.244025 21447 tablet_bootstrap.cc:492] T 29bae7c7b64244ce94e36360e740408d P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:44.245375 15605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29bae7c7b64244ce94e36360e740408d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:44.247684 20796 tablet_bootstrap.cc:492] T 29bae7c7b64244ce94e36360e740408d P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:44.248414 20796 tablet_bootstrap.cc:654] T 29bae7c7b64244ce94e36360e740408d P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:44.249506 20796 tablet_bootstrap.cc:492] T 29bae7c7b64244ce94e36360e740408d P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:44.249583 20796 ts_tablet_manager.cc:1367] T 29bae7c7b64244ce94e36360e740408d P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:56:44.249962 21447 tablet_bootstrap.cc:654] T 29bae7c7b64244ce94e36360e740408d P 61b887f0d40d497badeca5604c1f0546: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:44.249964 20796 raft_consensus.cc:348] T 29bae7c7b64244ce94e36360e740408d P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:44.250053 20796 raft_consensus.cc:374] T 29bae7c7b64244ce94e36360e740408d P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:44.250104 20796 raft_consensus.cc:729] T 29bae7c7b64244ce94e36360e740408d P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:44.250245 20796 consensus_queue.cc:260] T 29bae7c7b64244ce94e36360e740408d P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:44.250499 20796 ts_tablet_manager.cc:1398] T 29bae7c7b64244ce94e36360e740408d P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:44.251060 21448 tablet_bootstrap.cc:492] T 29bae7c7b64244ce94e36360e740408d P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:44.251194 21448 ts_tablet_manager.cc:1367] T 29bae7c7b64244ce94e36360e740408d P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 13:56:44.251658 21448 raft_consensus.cc:348] T 29bae7c7b64244ce94e36360e740408d P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:44.251765 21448 raft_consensus.cc:374] T 29bae7c7b64244ce94e36360e740408d P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:44.251791 21448 raft_consensus.cc:729] T 29bae7c7b64244ce94e36360e740408d P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:44.251904 21448 consensus_queue.cc:260] T 29bae7c7b64244ce94e36360e740408d P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:44.252164 21448 ts_tablet_manager.cc:1398] T 29bae7c7b64244ce94e36360e740408d P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:44.252322 21448 tablet_bootstrap.cc:492] T fd61c523ce9241ff8c2a649e037bbfc4 P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:44.254173 21447 tablet_bootstrap.cc:492] T 29bae7c7b64244ce94e36360e740408d P 61b887f0d40d497badeca5604c1f0546: No bootstrap required, opened a new log
I20240503 13:56:44.254321 21447 ts_tablet_manager.cc:1367] T 29bae7c7b64244ce94e36360e740408d P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.002s
I20240503 13:56:44.254796 21447 raft_consensus.cc:348] T 29bae7c7b64244ce94e36360e740408d P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:44.254907 21447 raft_consensus.cc:374] T 29bae7c7b64244ce94e36360e740408d P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:44.255831 21448 tablet_bootstrap.cc:654] T fd61c523ce9241ff8c2a649e037bbfc4 P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:44.255931 21447 raft_consensus.cc:729] T 29bae7c7b64244ce94e36360e740408d P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:44.256124 21447 consensus_queue.cc:260] T 29bae7c7b64244ce94e36360e740408d P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:44.256469 21447 ts_tablet_manager.cc:1398] T 29bae7c7b64244ce94e36360e740408d P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:56:44.258189 21447 tablet_bootstrap.cc:492] T 54e6b2f0d7da4d58a55d8a369f3843ce P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:44.259207 21447 tablet_bootstrap.cc:654] T 54e6b2f0d7da4d58a55d8a369f3843ce P 61b887f0d40d497badeca5604c1f0546: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:44.260572 21447 tablet_bootstrap.cc:492] T 54e6b2f0d7da4d58a55d8a369f3843ce P 61b887f0d40d497badeca5604c1f0546: No bootstrap required, opened a new log
I20240503 13:56:44.260671 21447 ts_tablet_manager.cc:1367] T 54e6b2f0d7da4d58a55d8a369f3843ce P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 13:56:44.261112 21447 raft_consensus.cc:348] T 54e6b2f0d7da4d58a55d8a369f3843ce P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:44.261219 21447 raft_consensus.cc:374] T 54e6b2f0d7da4d58a55d8a369f3843ce P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:44.261286 21447 raft_consensus.cc:729] T 54e6b2f0d7da4d58a55d8a369f3843ce P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:44.261436 21447 consensus_queue.cc:260] T 54e6b2f0d7da4d58a55d8a369f3843ce P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:44.261685 21447 ts_tablet_manager.cc:1398] T 54e6b2f0d7da4d58a55d8a369f3843ce P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:44.261947 21448 tablet_bootstrap.cc:492] T fd61c523ce9241ff8c2a649e037bbfc4 P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:44.262035 21448 ts_tablet_manager.cc:1367] T fd61c523ce9241ff8c2a649e037bbfc4 P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 13:56:44.262616 21448 raft_consensus.cc:348] T fd61c523ce9241ff8c2a649e037bbfc4 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:44.262732 21448 raft_consensus.cc:374] T fd61c523ce9241ff8c2a649e037bbfc4 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:44.262794 21448 raft_consensus.cc:729] T fd61c523ce9241ff8c2a649e037bbfc4 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:44.262923 21448 consensus_queue.cc:260] T fd61c523ce9241ff8c2a649e037bbfc4 P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:44.264235 21448 ts_tablet_manager.cc:1398] T fd61c523ce9241ff8c2a649e037bbfc4 P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:44.264459 21448 tablet_bootstrap.cc:492] T 54e6b2f0d7da4d58a55d8a369f3843ce P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:44.266311 21448 tablet_bootstrap.cc:654] T 54e6b2f0d7da4d58a55d8a369f3843ce P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:44.267539 21448 tablet_bootstrap.cc:492] T 54e6b2f0d7da4d58a55d8a369f3843ce P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:44.267628 21448 ts_tablet_manager.cc:1367] T 54e6b2f0d7da4d58a55d8a369f3843ce P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:56:44.268034 21448 raft_consensus.cc:348] T 54e6b2f0d7da4d58a55d8a369f3843ce P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:44.268146 21448 raft_consensus.cc:374] T 54e6b2f0d7da4d58a55d8a369f3843ce P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:44.268178 21448 raft_consensus.cc:729] T 54e6b2f0d7da4d58a55d8a369f3843ce P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:44.268291 21448 consensus_queue.cc:260] T 54e6b2f0d7da4d58a55d8a369f3843ce P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:44.268586 21448 ts_tablet_manager.cc:1398] T 54e6b2f0d7da4d58a55d8a369f3843ce P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:44.275218 21466 ts_tablet_manager.cc:902] T 397f0e3e877d4acd9157bdf99bee6255 P e47105a0f46e4511b7112e571688a4c7: Initiating tablet copy from peer ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:44.276177 21466 tablet_copy_client.cc:308] T 397f0e3e877d4acd9157bdf99bee6255 P e47105a0f46e4511b7112e571688a4c7: tablet copy: Beginning tablet copy session from remote peer at address 127.6.98.67:42279
I20240503 13:56:44.276643 13336 tablet_copy_service.cc:140] P ba0194098aa84449ae17e8a0d4c43703: Received BeginTabletCopySession request for tablet 397f0e3e877d4acd9157bdf99bee6255 from peer e47105a0f46e4511b7112e571688a4c7 ({username='slave'} at 127.6.98.72:58879)
I20240503 13:56:44.276741 13336 tablet_copy_service.cc:161] P ba0194098aa84449ae17e8a0d4c43703: Beginning new tablet copy session on tablet 397f0e3e877d4acd9157bdf99bee6255 from peer e47105a0f46e4511b7112e571688a4c7 at {username='slave'} at 127.6.98.72:58879: session id = e47105a0f46e4511b7112e571688a4c7-397f0e3e877d4acd9157bdf99bee6255
I20240503 13:56:44.277433 13336 tablet_copy_source_session.cc:205] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:56:44.278304 21466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 397f0e3e877d4acd9157bdf99bee6255. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:44.280947 21466 tablet_copy_client.cc:783] T 397f0e3e877d4acd9157bdf99bee6255 P e47105a0f46e4511b7112e571688a4c7: tablet copy: Starting download of 0 data blocks...
I20240503 13:56:44.281231 21466 tablet_copy_client.cc:647] T 397f0e3e877d4acd9157bdf99bee6255 P e47105a0f46e4511b7112e571688a4c7: tablet copy: Starting download of 1 WAL segments...
I20240503 13:56:44.283282 21466 tablet_copy_client.cc:523] T 397f0e3e877d4acd9157bdf99bee6255 P e47105a0f46e4511b7112e571688a4c7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:56:44.285828 21466 tablet_bootstrap.cc:492] T 397f0e3e877d4acd9157bdf99bee6255 P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:44.309653 21062 raft_consensus.cc:482] T 29bae7c7b64244ce94e36360e740408d P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:44.309787 21062 raft_consensus.cc:504] T 29bae7c7b64244ce94e36360e740408d P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:44.310113 20722 raft_consensus.cc:482] T 54e6b2f0d7da4d58a55d8a369f3843ce P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:44.310189 20722 raft_consensus.cc:504] T 54e6b2f0d7da4d58a55d8a369f3843ce P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:44.310288 21062 leader_election.cc:288] T 29bae7c7b64244ce94e36360e740408d P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951), 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885)
I20240503 13:56:44.310462 20722 leader_election.cc:288] T 54e6b2f0d7da4d58a55d8a369f3843ce P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885), 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:44.310626 15624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29bae7c7b64244ce94e36360e740408d" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d" is_pre_election: true
I20240503 13:56:44.310777 15624 raft_consensus.cc:2454] T 29bae7c7b64244ce94e36360e740408d P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 0.
I20240503 13:56:44.311100 15426 leader_election.cc:302] T 29bae7c7b64244ce94e36360e740408d P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 61b887f0d40d497badeca5604c1f0546; no voters:
I20240503 13:56:44.311277 20722 raft_consensus.cc:2781] T 29bae7c7b64244ce94e36360e740408d P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:44.311342 20722 raft_consensus.cc:482] T 29bae7c7b64244ce94e36360e740408d P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:44.311376 20722 raft_consensus.cc:3037] T 29bae7c7b64244ce94e36360e740408d P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:44.311733 15758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29bae7c7b64244ce94e36360e740408d" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2" is_pre_election: true
I20240503 13:56:44.311863 15758 raft_consensus.cc:2454] T 29bae7c7b64244ce94e36360e740408d P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 0.
I20240503 13:56:44.312016 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54e6b2f0d7da4d58a55d8a369f3843ce" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2" is_pre_election: true
I20240503 13:56:44.312144 15757 raft_consensus.cc:2454] T 54e6b2f0d7da4d58a55d8a369f3843ce P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 0.
I20240503 13:56:44.312366 13052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54e6b2f0d7da4d58a55d8a369f3843ce" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e" is_pre_election: true
I20240503 13:56:44.312465 20722 raft_consensus.cc:504] T 29bae7c7b64244ce94e36360e740408d P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:44.312762 13052 raft_consensus.cc:2454] T 54e6b2f0d7da4d58a55d8a369f3843ce P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 0.
I20240503 13:56:44.312820 20722 leader_election.cc:288] T 29bae7c7b64244ce94e36360e740408d P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 election: Requested vote from peers 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951), 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885)
I20240503 13:56:44.313102 15427 leader_election.cc:302] T 54e6b2f0d7da4d58a55d8a369f3843ce P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 61b887f0d40d497badeca5604c1f0546; no voters:
I20240503 13:56:44.313330 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29bae7c7b64244ce94e36360e740408d" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2"
I20240503 13:56:44.313335 15624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29bae7c7b64244ce94e36360e740408d" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d"
I20240503 13:56:44.313436 15757 raft_consensus.cc:3037] T 29bae7c7b64244ce94e36360e740408d P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:44.313442 15624 raft_consensus.cc:3037] T 29bae7c7b64244ce94e36360e740408d P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:44.313459 20722 raft_consensus.cc:2781] T 54e6b2f0d7da4d58a55d8a369f3843ce P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:44.313503 20722 raft_consensus.cc:482] T 54e6b2f0d7da4d58a55d8a369f3843ce P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:44.313532 20722 raft_consensus.cc:3037] T 54e6b2f0d7da4d58a55d8a369f3843ce P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:44.314384 15624 raft_consensus.cc:2454] T 29bae7c7b64244ce94e36360e740408d P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 1.
I20240503 13:56:44.314406 20722 raft_consensus.cc:504] T 54e6b2f0d7da4d58a55d8a369f3843ce P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:44.314656 15757 raft_consensus.cc:2454] T 29bae7c7b64244ce94e36360e740408d P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 1.
I20240503 13:56:44.315035 15426 leader_election.cc:302] T 29bae7c7b64244ce94e36360e740408d P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 61b887f0d40d497badeca5604c1f0546; no voters:
I20240503 13:56:44.315261 13052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54e6b2f0d7da4d58a55d8a369f3843ce" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e"
I20240503 13:56:44.315367 13052 raft_consensus.cc:3037] T 54e6b2f0d7da4d58a55d8a369f3843ce P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:44.316377 13052 raft_consensus.cc:2454] T 54e6b2f0d7da4d58a55d8a369f3843ce P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 1.
I20240503 13:56:44.316709 15427 leader_election.cc:302] T 54e6b2f0d7da4d58a55d8a369f3843ce P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 61b887f0d40d497badeca5604c1f0546; no voters:
I20240503 13:56:44.316872 21062 raft_consensus.cc:2781] T 29bae7c7b64244ce94e36360e740408d P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:44.317023 21062 raft_consensus.cc:686] T 29bae7c7b64244ce94e36360e740408d P 61b887f0d40d497badeca5604c1f0546 [term 1 LEADER]: Becoming Leader. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Running, Role: LEADER
I20240503 13:56:44.317122 21492 raft_consensus.cc:2781] T 54e6b2f0d7da4d58a55d8a369f3843ce P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:44.317204 21492 raft_consensus.cc:686] T 54e6b2f0d7da4d58a55d8a369f3843ce P 61b887f0d40d497badeca5604c1f0546 [term 1 LEADER]: Becoming Leader. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Running, Role: LEADER
I20240503 13:56:44.317168 21062 consensus_queue.cc:237] T 29bae7c7b64244ce94e36360e740408d P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:44.317322 21492 consensus_queue.cc:237] T 54e6b2f0d7da4d58a55d8a369f3843ce P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:44.317438 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54e6b2f0d7da4d58a55d8a369f3843ce" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2"
I20240503 13:56:44.317529 15757 raft_consensus.cc:3037] T 54e6b2f0d7da4d58a55d8a369f3843ce P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:44.318439 15757 raft_consensus.cc:2454] T 54e6b2f0d7da4d58a55d8a369f3843ce P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 1.
I20240503 13:56:44.318568 20722 leader_election.cc:288] T 54e6b2f0d7da4d58a55d8a369f3843ce P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 election: Requested vote from peers 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885), 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:44.319738 12904 catalog_manager.cc:5484] T 29bae7c7b64244ce94e36360e740408d P 61b887f0d40d497badeca5604c1f0546 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61b887f0d40d497badeca5604c1f0546 (127.6.98.69). New cstate: current_term: 1 leader_uuid: "61b887f0d40d497badeca5604c1f0546" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:44.322443 12905 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:56:44.325547 12903 catalog_manager.cc:5484] T 54e6b2f0d7da4d58a55d8a369f3843ce P 61b887f0d40d497badeca5604c1f0546 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61b887f0d40d497badeca5604c1f0546 (127.6.98.69). New cstate: current_term: 1 leader_uuid: "61b887f0d40d497badeca5604c1f0546" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } }
I20240503 13:56:44.328235 21435 tablet.cc:1722] T 0040894be67a49f5ab84eb0dc1f309a8 P 8ccf5a4ac450414c9d3579bd5800b770: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:44.328578 21435 tablet.cc:1620] T 0040894be67a49f5ab84eb0dc1f309a8 P 8ccf5a4ac450414c9d3579bd5800b770: MemRowSet was empty: no flush needed.
I20240503 13:56:44.328677 21441 tablet.cc:1722] T 0040894be67a49f5ab84eb0dc1f309a8 P ba0194098aa84449ae17e8a0d4c43703: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:44.328969 21441 tablet.cc:1620] T 0040894be67a49f5ab84eb0dc1f309a8 P ba0194098aa84449ae17e8a0d4c43703: MemRowSet was empty: no flush needed.
I20240503 13:56:44.328034 21466 tablet_bootstrap.cc:492] T 397f0e3e877d4acd9157bdf99bee6255 P e47105a0f46e4511b7112e571688a4c7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:56:44.337608 12905 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=ce0ba30e09fd4ea789864f5d1b8d2a1e] alter complete (version 2)
I20240503 13:56:44.340559 21436 tablet.cc:1722] T 0040894be67a49f5ab84eb0dc1f309a8 P 009f849ee7b54c80a3f5264695047e4d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:44.340886 21436 tablet.cc:1620] T 0040894be67a49f5ab84eb0dc1f309a8 P 009f849ee7b54c80a3f5264695047e4d: MemRowSet was empty: no flush needed.
I20240503 13:56:44.347203 21496 ts_tablet_manager.cc:902] T 356ccd6fdf25470dba2a79367a37128e P 8ccf5a4ac450414c9d3579bd5800b770: Initiating tablet copy from peer 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:44.348160 21496 tablet_copy_client.cc:308] T 356ccd6fdf25470dba2a79367a37128e P 8ccf5a4ac450414c9d3579bd5800b770: tablet copy: Beginning tablet copy session from remote peer at address 127.6.98.68:36109
I20240503 13:56:44.349156 13467 tablet_copy_service.cc:140] P 7690ee79577c45e3a2c8f8b552c0bff8: Received BeginTabletCopySession request for tablet 356ccd6fdf25470dba2a79367a37128e from peer 8ccf5a4ac450414c9d3579bd5800b770 ({username='slave'} at 127.6.98.66:43787)
I20240503 13:56:44.349269 13467 tablet_copy_service.cc:161] P 7690ee79577c45e3a2c8f8b552c0bff8: Beginning new tablet copy session on tablet 356ccd6fdf25470dba2a79367a37128e from peer 8ccf5a4ac450414c9d3579bd5800b770 at {username='slave'} at 127.6.98.66:43787: session id = 8ccf5a4ac450414c9d3579bd5800b770-356ccd6fdf25470dba2a79367a37128e
I20240503 13:56:44.350183 13467 tablet_copy_source_session.cc:205] T 356ccd6fdf25470dba2a79367a37128e P 7690ee79577c45e3a2c8f8b552c0bff8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:56:44.351835 21496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 356ccd6fdf25470dba2a79367a37128e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:44.352707 21466 tablet_bootstrap.cc:492] T 397f0e3e877d4acd9157bdf99bee6255 P e47105a0f46e4511b7112e571688a4c7: Bootstrap complete.
I20240503 13:56:44.352914 21466 ts_tablet_manager.cc:1367] T 397f0e3e877d4acd9157bdf99bee6255 P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.067s user 0.025s sys 0.000s
I20240503 13:56:44.353363 21466 raft_consensus.cc:348] T 397f0e3e877d4acd9157bdf99bee6255 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:44.353490 21466 raft_consensus.cc:729] T 397f0e3e877d4acd9157bdf99bee6255 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:44.353610 21466 consensus_queue.cc:260] T 397f0e3e877d4acd9157bdf99bee6255 P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:44.353905 21466 ts_tablet_manager.cc:1398] T 397f0e3e877d4acd9157bdf99bee6255 P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:44.354986 13336 tablet_copy_service.cc:342] P ba0194098aa84449ae17e8a0d4c43703: Request end of tablet copy session e47105a0f46e4511b7112e571688a4c7-397f0e3e877d4acd9157bdf99bee6255 received from {username='slave'} at 127.6.98.72:58879
I20240503 13:56:44.355082 13336 tablet_copy_service.cc:433] P ba0194098aa84449ae17e8a0d4c43703: ending tablet copy session e47105a0f46e4511b7112e571688a4c7-397f0e3e877d4acd9157bdf99bee6255 on tablet 397f0e3e877d4acd9157bdf99bee6255 with peer e47105a0f46e4511b7112e571688a4c7
I20240503 13:56:44.358070 21496 tablet_copy_client.cc:783] T 356ccd6fdf25470dba2a79367a37128e P 8ccf5a4ac450414c9d3579bd5800b770: tablet copy: Starting download of 0 data blocks...
I20240503 13:56:44.358286 21496 tablet_copy_client.cc:647] T 356ccd6fdf25470dba2a79367a37128e P 8ccf5a4ac450414c9d3579bd5800b770: tablet copy: Starting download of 1 WAL segments...
I20240503 13:56:44.360209 21496 tablet_copy_client.cc:523] T 356ccd6fdf25470dba2a79367a37128e P 8ccf5a4ac450414c9d3579bd5800b770: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:56:44.360656 21495 ts_tablet_manager.cc:902] T fcb0aa9fce544ae2a6f1ca96c5315c36 P ba0194098aa84449ae17e8a0d4c43703: Initiating tablet copy from peer 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885)
I20240503 13:56:44.361177 21495 tablet_copy_client.cc:308] T fcb0aa9fce544ae2a6f1ca96c5315c36 P ba0194098aa84449ae17e8a0d4c43703: tablet copy: Beginning tablet copy session from remote peer at address 127.6.98.71:42885
I20240503 13:56:44.361562 21496 tablet_bootstrap.cc:492] T 356ccd6fdf25470dba2a79367a37128e P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:44.361699 15778 tablet_copy_service.cc:140] P 76c386bef61a4bd5a907266a5118ffe2: Received BeginTabletCopySession request for tablet fcb0aa9fce544ae2a6f1ca96c5315c36 from peer ba0194098aa84449ae17e8a0d4c43703 ({username='slave'} at 127.6.98.67:56005)
I20240503 13:56:44.361801 15778 tablet_copy_service.cc:161] P 76c386bef61a4bd5a907266a5118ffe2: Beginning new tablet copy session on tablet fcb0aa9fce544ae2a6f1ca96c5315c36 from peer ba0194098aa84449ae17e8a0d4c43703 at {username='slave'} at 127.6.98.67:56005: session id = ba0194098aa84449ae17e8a0d4c43703-fcb0aa9fce544ae2a6f1ca96c5315c36
I20240503 13:56:44.362901 15778 tablet_copy_source_session.cc:205] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 76c386bef61a4bd5a907266a5118ffe2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:56:44.363898 21466 ts_tablet_manager.cc:902] T cbd6d38ea18b484d80f4a918148efd3a P e47105a0f46e4511b7112e571688a4c7: Initiating tablet copy from peer 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:44.364298 21466 tablet_copy_client.cc:308] T cbd6d38ea18b484d80f4a918148efd3a P e47105a0f46e4511b7112e571688a4c7: tablet copy: Beginning tablet copy session from remote peer at address 127.6.98.66:36387
I20240503 13:56:44.364713 13205 tablet_copy_service.cc:140] P 8ccf5a4ac450414c9d3579bd5800b770: Received BeginTabletCopySession request for tablet cbd6d38ea18b484d80f4a918148efd3a from peer e47105a0f46e4511b7112e571688a4c7 ({username='slave'} at 127.6.98.72:45181)
I20240503 13:56:44.364804 13205 tablet_copy_service.cc:161] P 8ccf5a4ac450414c9d3579bd5800b770: Beginning new tablet copy session on tablet cbd6d38ea18b484d80f4a918148efd3a from peer e47105a0f46e4511b7112e571688a4c7 at {username='slave'} at 127.6.98.72:45181: session id = e47105a0f46e4511b7112e571688a4c7-cbd6d38ea18b484d80f4a918148efd3a
I20240503 13:56:44.365200 21495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcb0aa9fce544ae2a6f1ca96c5315c36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:44.365707 13205 tablet_copy_source_session.cc:205] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:56:44.367136 21466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbd6d38ea18b484d80f4a918148efd3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:44.367501 21495 tablet_copy_client.cc:783] T fcb0aa9fce544ae2a6f1ca96c5315c36 P ba0194098aa84449ae17e8a0d4c43703: tablet copy: Starting download of 0 data blocks...
I20240503 13:56:44.367640 21495 tablet_copy_client.cc:647] T fcb0aa9fce544ae2a6f1ca96c5315c36 P ba0194098aa84449ae17e8a0d4c43703: tablet copy: Starting download of 1 WAL segments...
I20240503 13:56:44.368945 21496 tablet_bootstrap.cc:492] T 356ccd6fdf25470dba2a79367a37128e P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:56:44.369236 21496 tablet_bootstrap.cc:492] T 356ccd6fdf25470dba2a79367a37128e P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap complete.
I20240503 13:56:44.369474 21496 ts_tablet_manager.cc:1367] T 356ccd6fdf25470dba2a79367a37128e P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.001s
I20240503 13:56:44.369527 21466 tablet_copy_client.cc:783] T cbd6d38ea18b484d80f4a918148efd3a P e47105a0f46e4511b7112e571688a4c7: tablet copy: Starting download of 0 data blocks...
I20240503 13:56:44.369741 21466 tablet_copy_client.cc:647] T cbd6d38ea18b484d80f4a918148efd3a P e47105a0f46e4511b7112e571688a4c7: tablet copy: Starting download of 1 WAL segments...
I20240503 13:56:44.370048 21496 raft_consensus.cc:348] T 356ccd6fdf25470dba2a79367a37128e P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:44.370159 21495 tablet_copy_client.cc:523] T fcb0aa9fce544ae2a6f1ca96c5315c36 P ba0194098aa84449ae17e8a0d4c43703: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:56:44.370173 21496 raft_consensus.cc:729] T 356ccd6fdf25470dba2a79367a37128e P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:44.370327 21496 consensus_queue.cc:260] T 356ccd6fdf25470dba2a79367a37128e P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:44.370713 21496 ts_tablet_manager.cc:1398] T 356ccd6fdf25470dba2a79367a37128e P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:44.371884 13467 tablet_copy_service.cc:342] P 7690ee79577c45e3a2c8f8b552c0bff8: Request end of tablet copy session 8ccf5a4ac450414c9d3579bd5800b770-356ccd6fdf25470dba2a79367a37128e received from {username='slave'} at 127.6.98.66:43787
I20240503 13:56:44.372013 13467 tablet_copy_service.cc:433] P 7690ee79577c45e3a2c8f8b552c0bff8: ending tablet copy session 8ccf5a4ac450414c9d3579bd5800b770-356ccd6fdf25470dba2a79367a37128e on tablet 356ccd6fdf25470dba2a79367a37128e with peer 8ccf5a4ac450414c9d3579bd5800b770
I20240503 13:56:44.373548 21466 tablet_copy_client.cc:523] T cbd6d38ea18b484d80f4a918148efd3a P e47105a0f46e4511b7112e571688a4c7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:56:44.373572 21495 tablet_bootstrap.cc:492] T fcb0aa9fce544ae2a6f1ca96c5315c36 P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:44.375257 21466 tablet_bootstrap.cc:492] T cbd6d38ea18b484d80f4a918148efd3a P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:44.378281 21495 tablet_bootstrap.cc:492] T fcb0aa9fce544ae2a6f1ca96c5315c36 P ba0194098aa84449ae17e8a0d4c43703: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:56:44.378396 21466 tablet_bootstrap.cc:492] T cbd6d38ea18b484d80f4a918148efd3a P e47105a0f46e4511b7112e571688a4c7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:56:44.378530 21495 tablet_bootstrap.cc:492] T fcb0aa9fce544ae2a6f1ca96c5315c36 P ba0194098aa84449ae17e8a0d4c43703: Bootstrap complete.
I20240503 13:56:44.378608 21466 tablet_bootstrap.cc:492] T cbd6d38ea18b484d80f4a918148efd3a P e47105a0f46e4511b7112e571688a4c7: Bootstrap complete.
I20240503 13:56:44.378685 21495 ts_tablet_manager.cc:1367] T fcb0aa9fce544ae2a6f1ca96c5315c36 P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 13:56:44.378772 21466 ts_tablet_manager.cc:1367] T cbd6d38ea18b484d80f4a918148efd3a P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240503 13:56:44.379163 21495 raft_consensus.cc:348] T fcb0aa9fce544ae2a6f1ca96c5315c36 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:44.379134 21466 raft_consensus.cc:348] T cbd6d38ea18b484d80f4a918148efd3a P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:44.379272 21495 raft_consensus.cc:729] T fcb0aa9fce544ae2a6f1ca96c5315c36 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:44.379280 21466 raft_consensus.cc:729] T cbd6d38ea18b484d80f4a918148efd3a P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:44.379421 21466 consensus_queue.cc:260] T cbd6d38ea18b484d80f4a918148efd3a P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:44.379437 21495 consensus_queue.cc:260] T fcb0aa9fce544ae2a6f1ca96c5315c36 P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:44.379719 21495 ts_tablet_manager.cc:1398] T fcb0aa9fce544ae2a6f1ca96c5315c36 P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:44.380574 15778 tablet_copy_service.cc:342] P 76c386bef61a4bd5a907266a5118ffe2: Request end of tablet copy session ba0194098aa84449ae17e8a0d4c43703-fcb0aa9fce544ae2a6f1ca96c5315c36 received from {username='slave'} at 127.6.98.67:56005
I20240503 13:56:44.380673 15778 tablet_copy_service.cc:433] P 76c386bef61a4bd5a907266a5118ffe2: ending tablet copy session ba0194098aa84449ae17e8a0d4c43703-fcb0aa9fce544ae2a6f1ca96c5315c36 on tablet fcb0aa9fce544ae2a6f1ca96c5315c36 with peer ba0194098aa84449ae17e8a0d4c43703
I20240503 13:56:44.380674 21466 ts_tablet_manager.cc:1398] T cbd6d38ea18b484d80f4a918148efd3a P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:44.381605 13205 tablet_copy_service.cc:342] P 8ccf5a4ac450414c9d3579bd5800b770: Request end of tablet copy session e47105a0f46e4511b7112e571688a4c7-cbd6d38ea18b484d80f4a918148efd3a received from {username='slave'} at 127.6.98.72:45181
I20240503 13:56:44.381683 13205 tablet_copy_service.cc:433] P 8ccf5a4ac450414c9d3579bd5800b770: ending tablet copy session e47105a0f46e4511b7112e571688a4c7-cbd6d38ea18b484d80f4a918148efd3a on tablet cbd6d38ea18b484d80f4a918148efd3a with peer e47105a0f46e4511b7112e571688a4c7
I20240503 13:56:44.449092 12912 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:56:44.454540 21435 tablet.cc:1722] T 0040894be67a49f5ab84eb0dc1f309a8 P 8ccf5a4ac450414c9d3579bd5800b770: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:44.455118 21436 tablet.cc:1722] T 0040894be67a49f5ab84eb0dc1f309a8 P 009f849ee7b54c80a3f5264695047e4d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:44.455453 21441 tablet.cc:1722] T 0040894be67a49f5ab84eb0dc1f309a8 P ba0194098aa84449ae17e8a0d4c43703: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:44.459972 12903 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table_new_name [id=ce0ba30e09fd4ea789864f5d1b8d2a1e] alter complete (version 3)
I20240503 13:56:44.514492 13445 raft_consensus.cc:1203] T 6bca3e19362842fe928adb84e7c9297a P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:56:44.545493 15624 raft_consensus.cc:1203] T 023517b9d4ee43f497d98c4d7a6037df P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:56:44.564554 12912 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:44.564719 12912 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:44.566944 12912 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Sending DeleteTablet for 3 replicas of tablet 0040894be67a49f5ab84eb0dc1f309a8
I20240503 13:56:44.567529 13164 tablet_service.cc:1514] Processing DeleteTablet for tablet 0040894be67a49f5ab84eb0dc1f309a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:44 UTC) from {username='slave'} at 127.0.0.1:41668
I20240503 13:56:44.567946 15605 tablet_service.cc:1514] Processing DeleteTablet for tablet 0040894be67a49f5ab84eb0dc1f309a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:44 UTC) from {username='slave'} at 127.0.0.1:60718
I20240503 13:56:44.568198 13296 tablet_service.cc:1514] Processing DeleteTablet for tablet 0040894be67a49f5ab84eb0dc1f309a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:44 UTC) from {username='slave'} at 127.0.0.1:33250
I20240503 13:56:44.568310 21513 tablet_replica.cc:330] T 0040894be67a49f5ab84eb0dc1f309a8 P 009f849ee7b54c80a3f5264695047e4d: stopping tablet replica
I20240503 13:56:44.568449 21513 raft_consensus.cc:2229] T 0040894be67a49f5ab84eb0dc1f309a8 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:44.568639 21513 raft_consensus.cc:2258] T 0040894be67a49f5ab84eb0dc1f309a8 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:44.568733 21512 tablet_replica.cc:330] T 0040894be67a49f5ab84eb0dc1f309a8 P 8ccf5a4ac450414c9d3579bd5800b770: stopping tablet replica
I20240503 13:56:44.568897 21512 raft_consensus.cc:2229] T 0040894be67a49f5ab84eb0dc1f309a8 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:56:44.569147 21512 raft_consensus.cc:2258] T 0040894be67a49f5ab84eb0dc1f309a8 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:44.569341 21513 ts_tablet_manager.cc:1875] T 0040894be67a49f5ab84eb0dc1f309a8 P 009f849ee7b54c80a3f5264695047e4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:44.569784 21512 ts_tablet_manager.cc:1875] T 0040894be67a49f5ab84eb0dc1f309a8 P 8ccf5a4ac450414c9d3579bd5800b770: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:44.570573 21514 tablet_replica.cc:330] T 0040894be67a49f5ab84eb0dc1f309a8 P ba0194098aa84449ae17e8a0d4c43703: stopping tablet replica
I20240503 13:56:44.570823 21514 raft_consensus.cc:2229] T 0040894be67a49f5ab84eb0dc1f309a8 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:44.570995 21514 raft_consensus.cc:2258] T 0040894be67a49f5ab84eb0dc1f309a8 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:44.571915 21514 ts_tablet_manager.cc:1875] T 0040894be67a49f5ab84eb0dc1f309a8 P ba0194098aa84449ae17e8a0d4c43703: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:44.572304 21513 ts_tablet_manager.cc:1888] T 0040894be67a49f5ab84eb0dc1f309a8 P 009f849ee7b54c80a3f5264695047e4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:44.572408 21513 log.cc:1196] T 0040894be67a49f5ab84eb0dc1f309a8 P 009f849ee7b54c80a3f5264695047e4d: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/wal/wals/0040894be67a49f5ab84eb0dc1f309a8
I20240503 13:56:44.572722 21513 ts_tablet_manager.cc:1909] T 0040894be67a49f5ab84eb0dc1f309a8 P 009f849ee7b54c80a3f5264695047e4d: Deleting consensus metadata
I20240503 13:56:44.572727 21512 ts_tablet_manager.cc:1888] T 0040894be67a49f5ab84eb0dc1f309a8 P 8ccf5a4ac450414c9d3579bd5800b770: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:44.572897 21512 log.cc:1196] T 0040894be67a49f5ab84eb0dc1f309a8 P 8ccf5a4ac450414c9d3579bd5800b770: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/wal/wals/0040894be67a49f5ab84eb0dc1f309a8
I20240503 13:56:44.573321 21512 ts_tablet_manager.cc:1909] T 0040894be67a49f5ab84eb0dc1f309a8 P 8ccf5a4ac450414c9d3579bd5800b770: Deleting consensus metadata
I20240503 13:56:44.573534 12896 catalog_manager.cc:4830] TS 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951): tablet 0040894be67a49f5ab84eb0dc1f309a8 (table rebalancer_dynamic_table_new_name [id=ce0ba30e09fd4ea789864f5d1b8d2a1e]) successfully deleted
I20240503 13:56:44.574195 12899 catalog_manager.cc:4830] TS 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387): tablet 0040894be67a49f5ab84eb0dc1f309a8 (table rebalancer_dynamic_table_new_name [id=ce0ba30e09fd4ea789864f5d1b8d2a1e]) successfully deleted
I20240503 13:56:44.574658 21514 ts_tablet_manager.cc:1888] T 0040894be67a49f5ab84eb0dc1f309a8 P ba0194098aa84449ae17e8a0d4c43703: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:44.574957 21514 log.cc:1196] T 0040894be67a49f5ab84eb0dc1f309a8 P ba0194098aa84449ae17e8a0d4c43703: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/wal/wals/0040894be67a49f5ab84eb0dc1f309a8
I20240503 13:56:44.575248 21514 ts_tablet_manager.cc:1909] T 0040894be67a49f5ab84eb0dc1f309a8 P ba0194098aa84449ae17e8a0d4c43703: Deleting consensus metadata
I20240503 13:56:44.575819 12899 catalog_manager.cc:4830] TS ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279): tablet 0040894be67a49f5ab84eb0dc1f309a8 (table rebalancer_dynamic_table_new_name [id=ce0ba30e09fd4ea789864f5d1b8d2a1e]) successfully deleted
I20240503 13:56:44.646862 21191 raft_consensus.cc:482] T fd61c523ce9241ff8c2a649e037bbfc4 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:44.646988 21191 raft_consensus.cc:504] T fd61c523ce9241ff8c2a649e037bbfc4 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:44.647380 21191 leader_election.cc:288] T fd61c523ce9241ff8c2a649e037bbfc4 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695), 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885)
I20240503 13:56:44.647716 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd61c523ce9241ff8c2a649e037bbfc4" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2" is_pre_election: true
I20240503 13:56:44.647722 13052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd61c523ce9241ff8c2a649e037bbfc4" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e" is_pre_election: true
I20240503 13:56:44.647843 15757 raft_consensus.cc:2454] T fd61c523ce9241ff8c2a649e037bbfc4 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 0.
I20240503 13:56:44.647862 13052 raft_consensus.cc:2454] T fd61c523ce9241ff8c2a649e037bbfc4 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 0.
I20240503 13:56:44.648114 15561 leader_election.cc:302] T fd61c523ce9241ff8c2a649e037bbfc4 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 76c386bef61a4bd5a907266a5118ffe2; no voters:
I20240503 13:56:44.648279 21191 raft_consensus.cc:2781] T fd61c523ce9241ff8c2a649e037bbfc4 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:44.648339 21191 raft_consensus.cc:482] T fd61c523ce9241ff8c2a649e037bbfc4 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:44.648365 21191 raft_consensus.cc:3037] T fd61c523ce9241ff8c2a649e037bbfc4 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:44.649227 21191 raft_consensus.cc:504] T fd61c523ce9241ff8c2a649e037bbfc4 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:44.649541 21191 leader_election.cc:288] T fd61c523ce9241ff8c2a649e037bbfc4 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 election: Requested vote from peers 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695), 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885)
I20240503 13:56:44.649859 13052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd61c523ce9241ff8c2a649e037bbfc4" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e"
I20240503 13:56:44.649858 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd61c523ce9241ff8c2a649e037bbfc4" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2"
I20240503 13:56:44.650012 13052 raft_consensus.cc:3037] T fd61c523ce9241ff8c2a649e037bbfc4 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:44.649984 15757 raft_consensus.cc:3037] T fd61c523ce9241ff8c2a649e037bbfc4 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:44.650987 13052 raft_consensus.cc:2454] T fd61c523ce9241ff8c2a649e037bbfc4 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 1.
I20240503 13:56:44.650998 15757 raft_consensus.cc:2454] T fd61c523ce9241ff8c2a649e037bbfc4 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 1.
I20240503 13:56:44.651289 15559 leader_election.cc:302] T fd61c523ce9241ff8c2a649e037bbfc4 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 60ba3ffe3b7746469834cd91c84be14e; no voters:
I20240503 13:56:44.651472 21191 raft_consensus.cc:2781] T fd61c523ce9241ff8c2a649e037bbfc4 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:44.651648 21191 raft_consensus.cc:686] T fd61c523ce9241ff8c2a649e037bbfc4 P 009f849ee7b54c80a3f5264695047e4d [term 1 LEADER]: Becoming Leader. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Running, Role: LEADER
I20240503 13:56:44.651808 21191 consensus_queue.cc:237] T fd61c523ce9241ff8c2a649e037bbfc4 P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:44.653828 12912 catalog_manager.cc:5484] T fd61c523ce9241ff8c2a649e037bbfc4 P 009f849ee7b54c80a3f5264695047e4d reported cstate change: term changed from 0 to 1, leader changed from <none> to 009f849ee7b54c80a3f5264695047e4d (127.6.98.70). New cstate: current_term: 1 leader_uuid: "009f849ee7b54c80a3f5264695047e4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:44.669129 12912 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:56:44.674063 15471 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 13:56:44.675151 15605 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 13:56:44.677673 13164 tablet_service.cc:1467] Processing CreateTablet for tablet b8e0070d176c4b69ad2aca72bbb7c102 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c82e4bf13b5f46c6a7475628085ae2cf]), partition=
I20240503 13:56:44.677937 13164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8e0070d176c4b69ad2aca72bbb7c102. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:44.678659 13032 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:44.681277 15605 tablet_service.cc:1467] Processing CreateTablet for tablet b8e0070d176c4b69ad2aca72bbb7c102 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c82e4bf13b5f46c6a7475628085ae2cf]), partition=
I20240503 13:56:44.681545 15605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8e0070d176c4b69ad2aca72bbb7c102. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:44.683684 20796 tablet_bootstrap.cc:492] T b8e0070d176c4b69ad2aca72bbb7c102 P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:44.684561 20796 tablet_bootstrap.cc:654] T b8e0070d176c4b69ad2aca72bbb7c102 P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:44.686349 15738 tablet_service.cc:1467] Processing CreateTablet for tablet b8e0070d176c4b69ad2aca72bbb7c102 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c82e4bf13b5f46c6a7475628085ae2cf]), partition=
I20240503 13:56:44.686622 15738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8e0070d176c4b69ad2aca72bbb7c102. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:44.690495 21448 tablet_bootstrap.cc:492] T b8e0070d176c4b69ad2aca72bbb7c102 P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:44.691438 21448 tablet_bootstrap.cc:654] T b8e0070d176c4b69ad2aca72bbb7c102 P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:44.692281 13424 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:44.693142 21535 tablet_bootstrap.cc:492] T b8e0070d176c4b69ad2aca72bbb7c102 P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:44.694033 21535 tablet_bootstrap.cc:654] T b8e0070d176c4b69ad2aca72bbb7c102 P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:44.695520 21535 tablet_bootstrap.cc:492] T b8e0070d176c4b69ad2aca72bbb7c102 P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:44.695626 21535 ts_tablet_manager.cc:1367] T b8e0070d176c4b69ad2aca72bbb7c102 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:56:44.697134 15868 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:56:44.696382 21535 raft_consensus.cc:348] T b8e0070d176c4b69ad2aca72bbb7c102 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:44.697444 21535 raft_consensus.cc:374] T b8e0070d176c4b69ad2aca72bbb7c102 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:44.698889 21535 raft_consensus.cc:729] T b8e0070d176c4b69ad2aca72bbb7c102 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:44.699038 21535 consensus_queue.cc:260] T b8e0070d176c4b69ad2aca72bbb7c102 P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:44.700016 21535 ts_tablet_manager.cc:1398] T b8e0070d176c4b69ad2aca72bbb7c102 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 13:56:44.705377 13164 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 13:56:44.710399 20796 tablet_bootstrap.cc:492] T b8e0070d176c4b69ad2aca72bbb7c102 P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:44.710501 20796 ts_tablet_manager.cc:1367] T b8e0070d176c4b69ad2aca72bbb7c102 P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.027s user 0.000s sys 0.002s
I20240503 13:56:44.710938 20796 raft_consensus.cc:348] T b8e0070d176c4b69ad2aca72bbb7c102 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:44.711045 20796 raft_consensus.cc:374] T b8e0070d176c4b69ad2aca72bbb7c102 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:44.711072 20796 raft_consensus.cc:729] T b8e0070d176c4b69ad2aca72bbb7c102 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:44.711174 20796 consensus_queue.cc:260] T b8e0070d176c4b69ad2aca72bbb7c102 P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:44.711436 20796 ts_tablet_manager.cc:1398] T b8e0070d176c4b69ad2aca72bbb7c102 P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:44.714898 20722 consensus_queue.cc:1035] T 29bae7c7b64244ce94e36360e740408d P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:44.716627 13296 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 13:56:44.720410 21062 consensus_queue.cc:1035] T 29bae7c7b64244ce94e36360e740408d P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:44.724900 21448 tablet_bootstrap.cc:492] T b8e0070d176c4b69ad2aca72bbb7c102 P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:44.725015 21448 ts_tablet_manager.cc:1367] T b8e0070d176c4b69ad2aca72bbb7c102 P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.035s user 0.002s sys 0.000s
I20240503 13:56:44.725543 21448 raft_consensus.cc:348] T b8e0070d176c4b69ad2aca72bbb7c102 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:44.725656 21448 raft_consensus.cc:374] T b8e0070d176c4b69ad2aca72bbb7c102 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:44.725689 21448 raft_consensus.cc:729] T b8e0070d176c4b69ad2aca72bbb7c102 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:44.725800 21448 consensus_queue.cc:260] T b8e0070d176c4b69ad2aca72bbb7c102 P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:44.726102 21448 ts_tablet_manager.cc:1398] T b8e0070d176c4b69ad2aca72bbb7c102 P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:44.732715 15738 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 13:56:44.765517 19618 raft_consensus.cc:482] T b8e0070d176c4b69ad2aca72bbb7c102 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:44.765655 19618 raft_consensus.cc:504] T b8e0070d176c4b69ad2aca72bbb7c102 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:44.766166 19618 leader_election.cc:288] T b8e0070d176c4b69ad2aca72bbb7c102 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387), 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951)
I20240503 13:56:44.766572 15624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8e0070d176c4b69ad2aca72bbb7c102" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d" is_pre_election: true
I20240503 13:56:44.766705 15624 raft_consensus.cc:2454] T b8e0070d176c4b69ad2aca72bbb7c102 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 0.
I20240503 13:56:44.767189 13183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8e0070d176c4b69ad2aca72bbb7c102" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770" is_pre_election: true
I20240503 13:56:44.767313 13183 raft_consensus.cc:2454] T b8e0070d176c4b69ad2aca72bbb7c102 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 0.
I20240503 13:56:44.767292 15691 leader_election.cc:302] T b8e0070d176c4b69ad2aca72bbb7c102 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 76c386bef61a4bd5a907266a5118ffe2; no voters:
I20240503 13:56:44.769461 19618 raft_consensus.cc:2781] T b8e0070d176c4b69ad2aca72bbb7c102 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:44.769522 19618 raft_consensus.cc:482] T b8e0070d176c4b69ad2aca72bbb7c102 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:44.769558 19618 raft_consensus.cc:3037] T b8e0070d176c4b69ad2aca72bbb7c102 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:44.770432 19618 raft_consensus.cc:504] T b8e0070d176c4b69ad2aca72bbb7c102 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:44.770781 19618 leader_election.cc:288] T b8e0070d176c4b69ad2aca72bbb7c102 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 election: Requested vote from peers 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387), 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951)
I20240503 13:56:44.771258 13185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8e0070d176c4b69ad2aca72bbb7c102" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770"
I20240503 13:56:44.771441 13185 raft_consensus.cc:3037] T b8e0070d176c4b69ad2aca72bbb7c102 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:44.771458 15624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8e0070d176c4b69ad2aca72bbb7c102" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d"
I20240503 13:56:44.771606 15624 raft_consensus.cc:3037] T b8e0070d176c4b69ad2aca72bbb7c102 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:44.772377 13185 raft_consensus.cc:2454] T b8e0070d176c4b69ad2aca72bbb7c102 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 1.
I20240503 13:56:44.772473 15624 raft_consensus.cc:2454] T b8e0070d176c4b69ad2aca72bbb7c102 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 1.
I20240503 13:56:44.772740 15694 leader_election.cc:302] T b8e0070d176c4b69ad2aca72bbb7c102 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76c386bef61a4bd5a907266a5118ffe2, 8ccf5a4ac450414c9d3579bd5800b770; no voters:
I20240503 13:56:44.772948 21252 raft_consensus.cc:2781] T b8e0070d176c4b69ad2aca72bbb7c102 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:44.773042 21252 raft_consensus.cc:686] T b8e0070d176c4b69ad2aca72bbb7c102 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 LEADER]: Becoming Leader. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Running, Role: LEADER
I20240503 13:56:44.773214 21252 consensus_queue.cc:237] T b8e0070d176c4b69ad2aca72bbb7c102 P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:44.776526 12903 catalog_manager.cc:5484] T b8e0070d176c4b69ad2aca72bbb7c102 P 76c386bef61a4bd5a907266a5118ffe2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71). New cstate: current_term: 1 leader_uuid: "76c386bef61a4bd5a907266a5118ffe2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:44.796121 21062 consensus_queue.cc:1035] T 54e6b2f0d7da4d58a55d8a369f3843ce P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:44.815003 20722 consensus_queue.cc:1035] T 54e6b2f0d7da4d58a55d8a369f3843ce P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:44.819625 13185 raft_consensus.cc:1203] T 356ccd6fdf25470dba2a79367a37128e P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:56:44.822681 13316 raft_consensus.cc:1203] T fcb0aa9fce544ae2a6f1ca96c5315c36 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:56:44.879881 15471 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 13:56:44.880422 15888 raft_consensus.cc:1203] T cbd6d38ea18b484d80f4a918148efd3a P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:56:44.885469 12912 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:56:44.888340 15738 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 13:56:44.889137 13032 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:44.892170 13424 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:44.892180 15888 raft_consensus.cc:1203] T 397f0e3e877d4acd9157bdf99bee6255 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:56:44.896668 13296 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 13:56:44.899144 15624 raft_consensus.cc:1261] T b8e0070d176c4b69ad2aca72bbb7c102 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Refusing update from remote peer 76c386bef61a4bd5a907266a5118ffe2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:44.899751 13185 raft_consensus.cc:1261] T b8e0070d176c4b69ad2aca72bbb7c102 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Refusing update from remote peer 76c386bef61a4bd5a907266a5118ffe2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:44.900242 21252 consensus_queue.cc:1035] T b8e0070d176c4b69ad2aca72bbb7c102 P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:44.901224 13164 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 13:56:44.902640 19618 consensus_queue.cc:1035] T b8e0070d176c4b69ad2aca72bbb7c102 P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:44.910315 21436 tablet.cc:1722] T b8e0070d176c4b69ad2aca72bbb7c102 P 009f849ee7b54c80a3f5264695047e4d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:44.910668 21436 tablet.cc:1620] T b8e0070d176c4b69ad2aca72bbb7c102 P 009f849ee7b54c80a3f5264695047e4d: MemRowSet was empty: no flush needed.
I20240503 13:56:44.916958 12912 catalog_manager.cc:5576] TS 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951) does not have the latest schema for tablet b8e0070d176c4b69ad2aca72bbb7c102 (table rebalancer_dynamic_table [id=c82e4bf13b5f46c6a7475628085ae2cf]). Expected version 1 got 0
I20240503 13:56:44.922602 12904 catalog_manager.cc:5576] TS 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387) does not have the latest schema for tablet b8e0070d176c4b69ad2aca72bbb7c102 (table rebalancer_dynamic_table [id=c82e4bf13b5f46c6a7475628085ae2cf]). Expected version 1 got 0
I20240503 13:56:44.923182 12912 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=c82e4bf13b5f46c6a7475628085ae2cf] alter complete (version 1)
I20240503 13:56:44.932123 21602 tablet.cc:1722] T b8e0070d176c4b69ad2aca72bbb7c102 P 76c386bef61a4bd5a907266a5118ffe2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:44.933635 21435 tablet.cc:1722] T b8e0070d176c4b69ad2aca72bbb7c102 P 8ccf5a4ac450414c9d3579bd5800b770: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:44.933972 21435 tablet.cc:1620] T b8e0070d176c4b69ad2aca72bbb7c102 P 8ccf5a4ac450414c9d3579bd5800b770: MemRowSet was empty: no flush needed.
I20240503 13:56:44.934561 21602 tablet.cc:1620] T b8e0070d176c4b69ad2aca72bbb7c102 P 76c386bef61a4bd5a907266a5118ffe2: MemRowSet was empty: no flush needed.
I20240503 13:56:44.953366 15605 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 13:56:44.953632 15868 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:56:45.001602 13052 consensus_queue.cc:237] T 023517b9d4ee43f497d98c4d7a6037df P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:45.002753 13185 raft_consensus.cc:1261] T 023517b9d4ee43f497d98c4d7a6037df P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Refusing update from remote peer 60ba3ffe3b7746469834cd91c84be14e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:56:45.002857 15624 raft_consensus.cc:1261] T 023517b9d4ee43f497d98c4d7a6037df P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Refusing update from remote peer 60ba3ffe3b7746469834cd91c84be14e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:56:45.003218 20870 consensus_queue.cc:1035] T 023517b9d4ee43f497d98c4d7a6037df P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:45.003497 21211 consensus_queue.cc:1035] T 023517b9d4ee43f497d98c4d7a6037df P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:45.004809 20870 raft_consensus.cc:2932] T 023517b9d4ee43f497d98c4d7a6037df P 60ba3ffe3b7746469834cd91c84be14e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:45.005374 13185 raft_consensus.cc:2932] T 023517b9d4ee43f497d98c4d7a6037df P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:45.006465 15624 raft_consensus.cc:2932] T 023517b9d4ee43f497d98c4d7a6037df P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:45.008651 12904 catalog_manager.cc:5484] T 023517b9d4ee43f497d98c4d7a6037df P 8ccf5a4ac450414c9d3579bd5800b770 reported cstate change: config changed from index 4 to 5, VOTER ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) evicted. New cstate: current_term: 1 leader_uuid: "60ba3ffe3b7746469834cd91c84be14e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:45.010973 13296 tablet_service.cc:1514] Processing DeleteTablet for tablet 023517b9d4ee43f497d98c4d7a6037df with delete_type TABLET_DATA_TOMBSTONED (TS ba0194098aa84449ae17e8a0d4c43703 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:33250
I20240503 13:56:45.011212 21514 tablet_replica.cc:330] T 023517b9d4ee43f497d98c4d7a6037df P ba0194098aa84449ae17e8a0d4c43703: stopping tablet replica
I20240503 13:56:45.011351 21514 raft_consensus.cc:2229] T 023517b9d4ee43f497d98c4d7a6037df P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:45.011488 21514 raft_consensus.cc:2258] T 023517b9d4ee43f497d98c4d7a6037df P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:45.012010 21514 ts_tablet_manager.cc:1875] T 023517b9d4ee43f497d98c4d7a6037df P ba0194098aa84449ae17e8a0d4c43703: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:56:45.014147 21514 ts_tablet_manager.cc:1888] T 023517b9d4ee43f497d98c4d7a6037df P ba0194098aa84449ae17e8a0d4c43703: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 13:56:45.014210 21514 log.cc:1196] T 023517b9d4ee43f497d98c4d7a6037df P ba0194098aa84449ae17e8a0d4c43703: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/wal/wals/023517b9d4ee43f497d98c4d7a6037df
I20240503 13:56:45.014654 12899 catalog_manager.cc:4830] TS ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279): tablet 023517b9d4ee43f497d98c4d7a6037df (table rebalance_test_table_17 [id=52f6b247496e4c3eb63eef4f16160012]) successfully deleted
I20240503 13:56:45.031832 12905 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:56:45.037570 21602 tablet.cc:1722] T b8e0070d176c4b69ad2aca72bbb7c102 P 76c386bef61a4bd5a907266a5118ffe2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:45.037894 21602 tablet.cc:1620] T b8e0070d176c4b69ad2aca72bbb7c102 P 76c386bef61a4bd5a907266a5118ffe2: MemRowSet was empty: no flush needed.
I20240503 13:56:45.039664 21435 tablet.cc:1722] T b8e0070d176c4b69ad2aca72bbb7c102 P 8ccf5a4ac450414c9d3579bd5800b770: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:45.039988 21435 tablet.cc:1620] T b8e0070d176c4b69ad2aca72bbb7c102 P 8ccf5a4ac450414c9d3579bd5800b770: MemRowSet was empty: no flush needed.
I20240503 13:56:45.040705 21436 tablet.cc:1722] T b8e0070d176c4b69ad2aca72bbb7c102 P 009f849ee7b54c80a3f5264695047e4d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:45.040985 21436 tablet.cc:1620] T b8e0070d176c4b69ad2aca72bbb7c102 P 009f849ee7b54c80a3f5264695047e4d: MemRowSet was empty: no flush needed.
I20240503 13:56:45.045347 12905 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=c82e4bf13b5f46c6a7475628085ae2cf] alter complete (version 2)
I20240503 13:56:45.121187 13032 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:45.123309 15471 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 13:56:45.123669 15605 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 13:56:45.130591 15868 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:56:45.132302 13424 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:56:45.141559 15738 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 13:56:45.144805 13164 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 13:56:45.148808 12911 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:56:45.156014 21602 tablet.cc:1722] T b8e0070d176c4b69ad2aca72bbb7c102 P 76c386bef61a4bd5a907266a5118ffe2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:45.158650 21436 tablet.cc:1722] T b8e0070d176c4b69ad2aca72bbb7c102 P 009f849ee7b54c80a3f5264695047e4d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:45.166393 21435 tablet.cc:1722] T b8e0070d176c4b69ad2aca72bbb7c102 P 8ccf5a4ac450414c9d3579bd5800b770: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:45.171375 12911 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table_new_name [id=c82e4bf13b5f46c6a7475628085ae2cf] alter complete (version 3)
I20240503 13:56:45.179666 13296 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 13:56:45.191918 21515 consensus_queue.cc:1035] T fd61c523ce9241ff8c2a649e037bbfc4 P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:45.198287 21191 consensus_queue.cc:1035] T fd61c523ce9241ff8c2a649e037bbfc4 P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:45.209335 12911 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_extra_table_42"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:56:45.224198 15605 tablet_service.cc:1467] Processing CreateTablet for tablet 8438207c32194339a43e203f751d2345 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=960246928ff54227a7076d1099900636]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:45.224253 15868 tablet_service.cc:1467] Processing CreateTablet for tablet 8438207c32194339a43e203f751d2345 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=960246928ff54227a7076d1099900636]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:45.224537 15605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8438207c32194339a43e203f751d2345. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:45.224560 15868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8438207c32194339a43e203f751d2345. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:45.225196 15472 tablet_service.cc:1467] Processing CreateTablet for tablet abc88caa63f34af48f92e81918cf1d4e (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=960246928ff54227a7076d1099900636]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:45.225217 13032 tablet_service.cc:1467] Processing CreateTablet for tablet 8438207c32194339a43e203f751d2345 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=960246928ff54227a7076d1099900636]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:45.225447 15472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abc88caa63f34af48f92e81918cf1d4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:45.225466 13032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8438207c32194339a43e203f751d2345. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:45.227267 15868 tablet_service.cc:1467] Processing CreateTablet for tablet 9dd22ebd569c4d1da07bcb3365973406 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=960246928ff54227a7076d1099900636]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:45.227478 15868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dd22ebd569c4d1da07bcb3365973406. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:45.227766 15471 tablet_service.cc:1467] Processing CreateTablet for tablet 9dd22ebd569c4d1da07bcb3365973406 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=960246928ff54227a7076d1099900636]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:45.228005 15471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dd22ebd569c4d1da07bcb3365973406. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:45.228067 13424 tablet_service.cc:1467] Processing CreateTablet for tablet abc88caa63f34af48f92e81918cf1d4e (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=960246928ff54227a7076d1099900636]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:45.228302 13424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abc88caa63f34af48f92e81918cf1d4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:45.228914 15738 tablet_service.cc:1467] Processing CreateTablet for tablet abc88caa63f34af48f92e81918cf1d4e (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=960246928ff54227a7076d1099900636]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:45.229162 15738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abc88caa63f34af48f92e81918cf1d4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:45.230450 21684 tablet_bootstrap.cc:492] T abc88caa63f34af48f92e81918cf1d4e P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:45.231495 21684 tablet_bootstrap.cc:654] T abc88caa63f34af48f92e81918cf1d4e P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:45.231699 21685 tablet_bootstrap.cc:492] T abc88caa63f34af48f92e81918cf1d4e P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:45.232758 21685 tablet_bootstrap.cc:654] T abc88caa63f34af48f92e81918cf1d4e P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:45.232883 21684 tablet_bootstrap.cc:492] T abc88caa63f34af48f92e81918cf1d4e P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:45.232991 21684 ts_tablet_manager.cc:1367] T abc88caa63f34af48f92e81918cf1d4e P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:56:45.233438 21684 raft_consensus.cc:348] T abc88caa63f34af48f92e81918cf1d4e P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:45.233541 21684 raft_consensus.cc:374] T abc88caa63f34af48f92e81918cf1d4e P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:45.233570 21684 raft_consensus.cc:729] T abc88caa63f34af48f92e81918cf1d4e P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:45.233697 21684 consensus_queue.cc:260] T abc88caa63f34af48f92e81918cf1d4e P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:45.233947 21685 tablet_bootstrap.cc:492] T abc88caa63f34af48f92e81918cf1d4e P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:45.233991 21684 ts_tablet_manager.cc:1398] T abc88caa63f34af48f92e81918cf1d4e P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:45.234016 21685 ts_tablet_manager.cc:1367] T abc88caa63f34af48f92e81918cf1d4e P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:56:45.234393 21685 raft_consensus.cc:348] T abc88caa63f34af48f92e81918cf1d4e P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:45.234480 21685 raft_consensus.cc:374] T abc88caa63f34af48f92e81918cf1d4e P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:45.234505 21685 raft_consensus.cc:729] T abc88caa63f34af48f92e81918cf1d4e P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:45.234520 15605 tablet_service.cc:1467] Processing CreateTablet for tablet 9dd22ebd569c4d1da07bcb3365973406 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=960246928ff54227a7076d1099900636]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:45.234611 21685 consensus_queue.cc:260] T abc88caa63f34af48f92e81918cf1d4e P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:45.234758 15605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dd22ebd569c4d1da07bcb3365973406. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:45.234851 21685 ts_tablet_manager.cc:1398] T abc88caa63f34af48f92e81918cf1d4e P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:45.238142 21678 tablet_bootstrap.cc:492] T abc88caa63f34af48f92e81918cf1d4e P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:45.239358 21678 tablet_bootstrap.cc:654] T abc88caa63f34af48f92e81918cf1d4e P 61b887f0d40d497badeca5604c1f0546: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:45.241415 21678 tablet_bootstrap.cc:492] T abc88caa63f34af48f92e81918cf1d4e P 61b887f0d40d497badeca5604c1f0546: No bootstrap required, opened a new log
I20240503 13:56:45.241508 21678 ts_tablet_manager.cc:1367] T abc88caa63f34af48f92e81918cf1d4e P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:56:45.241915 21678 raft_consensus.cc:348] T abc88caa63f34af48f92e81918cf1d4e P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:45.242014 21678 raft_consensus.cc:374] T abc88caa63f34af48f92e81918cf1d4e P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:45.242041 21678 raft_consensus.cc:729] T abc88caa63f34af48f92e81918cf1d4e P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:45.242161 21679 tablet_bootstrap.cc:492] T 8438207c32194339a43e203f751d2345 P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:45.242209 21678 consensus_queue.cc:260] T abc88caa63f34af48f92e81918cf1d4e P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:45.242501 21678 ts_tablet_manager.cc:1398] T abc88caa63f34af48f92e81918cf1d4e P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:45.243259 21678 tablet_bootstrap.cc:492] T 9dd22ebd569c4d1da07bcb3365973406 P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:45.243397 21677 tablet_bootstrap.cc:492] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:45.244864 21679 tablet_bootstrap.cc:654] T 8438207c32194339a43e203f751d2345 P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:45.245651 21676 tablet_bootstrap.cc:492] T 8438207c32194339a43e203f751d2345 P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:45.246259 21678 tablet_bootstrap.cc:654] T 9dd22ebd569c4d1da07bcb3365973406 P 61b887f0d40d497badeca5604c1f0546: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:45.246260 21679 tablet_bootstrap.cc:492] T 8438207c32194339a43e203f751d2345 P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:45.246345 21679 ts_tablet_manager.cc:1367] T 8438207c32194339a43e203f751d2345 P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240503 13:56:45.246524 21676 tablet_bootstrap.cc:654] T 8438207c32194339a43e203f751d2345 P e47105a0f46e4511b7112e571688a4c7: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:45.244359 21677 tablet_bootstrap.cc:654] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:45.246953 21679 raft_consensus.cc:348] T 8438207c32194339a43e203f751d2345 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:45.247107 21679 raft_consensus.cc:374] T 8438207c32194339a43e203f751d2345 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:45.247155 21679 raft_consensus.cc:729] T 8438207c32194339a43e203f751d2345 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:45.247298 21679 consensus_queue.cc:260] T 8438207c32194339a43e203f751d2345 P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:45.248219 21678 tablet_bootstrap.cc:492] T 9dd22ebd569c4d1da07bcb3365973406 P 61b887f0d40d497badeca5604c1f0546: No bootstrap required, opened a new log
I20240503 13:56:45.248339 21678 ts_tablet_manager.cc:1367] T 9dd22ebd569c4d1da07bcb3365973406 P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:56:45.248368 21679 ts_tablet_manager.cc:1398] T 8438207c32194339a43e203f751d2345 P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:56:45.248816 21678 raft_consensus.cc:348] T 9dd22ebd569c4d1da07bcb3365973406 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:45.248927 21678 raft_consensus.cc:374] T 9dd22ebd569c4d1da07bcb3365973406 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:45.248997 21678 raft_consensus.cc:729] T 9dd22ebd569c4d1da07bcb3365973406 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:45.249313 21678 consensus_queue.cc:260] T 9dd22ebd569c4d1da07bcb3365973406 P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:45.249656 21678 ts_tablet_manager.cc:1398] T 9dd22ebd569c4d1da07bcb3365973406 P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:45.250552 21677 tablet_bootstrap.cc:492] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:45.251473 21677 ts_tablet_manager.cc:1367] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 13:56:45.251983 21677 raft_consensus.cc:348] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:45.252532 21676 tablet_bootstrap.cc:492] T 8438207c32194339a43e203f751d2345 P e47105a0f46e4511b7112e571688a4c7: No bootstrap required, opened a new log
I20240503 13:56:45.252664 21676 ts_tablet_manager.cc:1367] T 8438207c32194339a43e203f751d2345 P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 13:56:45.253149 21676 raft_consensus.cc:348] T 8438207c32194339a43e203f751d2345 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:45.253245 21676 raft_consensus.cc:374] T 8438207c32194339a43e203f751d2345 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:45.253273 21676 raft_consensus.cc:729] T 8438207c32194339a43e203f751d2345 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:45.253336 21677 raft_consensus.cc:374] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:45.253384 21677 raft_consensus.cc:729] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:45.253515 21677 consensus_queue.cc:260] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:45.253873 21677 ts_tablet_manager.cc:1398] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:45.256700 21676 consensus_queue.cc:260] T 8438207c32194339a43e203f751d2345 P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:45.258373 21677 tablet_bootstrap.cc:492] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:45.259092 21676 ts_tablet_manager.cc:1398] T 8438207c32194339a43e203f751d2345 P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240503 13:56:45.260057 21676 tablet_bootstrap.cc:492] T 9dd22ebd569c4d1da07bcb3365973406 P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:45.260753 21677 tablet_bootstrap.cc:654] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:45.261670 21676 tablet_bootstrap.cc:654] T 9dd22ebd569c4d1da07bcb3365973406 P e47105a0f46e4511b7112e571688a4c7: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:45.263494 21676 tablet_bootstrap.cc:492] T 9dd22ebd569c4d1da07bcb3365973406 P e47105a0f46e4511b7112e571688a4c7: No bootstrap required, opened a new log
I20240503 13:56:45.263585 21676 ts_tablet_manager.cc:1367] T 9dd22ebd569c4d1da07bcb3365973406 P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 13:56:45.264087 21676 raft_consensus.cc:348] T 9dd22ebd569c4d1da07bcb3365973406 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:45.264204 21676 raft_consensus.cc:374] T 9dd22ebd569c4d1da07bcb3365973406 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:45.264236 21676 raft_consensus.cc:729] T 9dd22ebd569c4d1da07bcb3365973406 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:45.264355 21676 consensus_queue.cc:260] T 9dd22ebd569c4d1da07bcb3365973406 P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:45.264678 21676 ts_tablet_manager.cc:1398] T 9dd22ebd569c4d1da07bcb3365973406 P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:45.269161 21677 tablet_bootstrap.cc:492] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:45.269255 21677 ts_tablet_manager.cc:1367] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 13:56:45.269696 21677 raft_consensus.cc:348] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:45.269790 21677 raft_consensus.cc:374] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:45.269825 21677 raft_consensus.cc:729] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:45.269940 21677 consensus_queue.cc:260] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:45.270193 21677 ts_tablet_manager.cc:1398] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:45.275614 12904 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:45.275758 12904 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:45.277503 20120 raft_consensus.cc:482] T abc88caa63f34af48f92e81918cf1d4e P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:45.277611 20120 raft_consensus.cc:504] T abc88caa63f34af48f92e81918cf1d4e P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:45.278084 20120 leader_election.cc:288] T abc88caa63f34af48f92e81918cf1d4e P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885), 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153)
I20240503 13:56:45.278381 12904 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Sending DeleteTablet for 3 replicas of tablet b8e0070d176c4b69ad2aca72bbb7c102
I20240503 13:56:45.278443 15758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abc88caa63f34af48f92e81918cf1d4e" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2" is_pre_election: true
I20240503 13:56:45.278594 15758 raft_consensus.cc:2454] T abc88caa63f34af48f92e81918cf1d4e P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 0.
I20240503 13:56:45.278504 15493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abc88caa63f34af48f92e81918cf1d4e" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546" is_pre_election: true
I20240503 13:56:45.278955 15493 raft_consensus.cc:2454] T abc88caa63f34af48f92e81918cf1d4e P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 0.
I20240503 13:56:45.279155 13383 leader_election.cc:302] T abc88caa63f34af48f92e81918cf1d4e P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, 76c386bef61a4bd5a907266a5118ffe2; no voters:
I20240503 13:56:45.279336 20120 raft_consensus.cc:2781] T abc88caa63f34af48f92e81918cf1d4e P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:45.279335 15605 tablet_service.cc:1514] Processing DeleteTablet for tablet b8e0070d176c4b69ad2aca72bbb7c102 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:45 UTC) from {username='slave'} at 127.0.0.1:60718
I20240503 13:56:45.279393 20120 raft_consensus.cc:482] T abc88caa63f34af48f92e81918cf1d4e P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:45.279435 20120 raft_consensus.cc:3037] T abc88caa63f34af48f92e81918cf1d4e P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:45.279614 15738 tablet_service.cc:1514] Processing DeleteTablet for tablet b8e0070d176c4b69ad2aca72bbb7c102 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:45 UTC) from {username='slave'} at 127.0.0.1:44984
I20240503 13:56:45.279641 21706 tablet_replica.cc:330] T b8e0070d176c4b69ad2aca72bbb7c102 P 009f849ee7b54c80a3f5264695047e4d: stopping tablet replica
I20240503 13:56:45.279771 21706 raft_consensus.cc:2229] T b8e0070d176c4b69ad2aca72bbb7c102 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:45.279889 21707 tablet_replica.cc:330] T b8e0070d176c4b69ad2aca72bbb7c102 P 76c386bef61a4bd5a907266a5118ffe2: stopping tablet replica
I20240503 13:56:45.279927 21706 raft_consensus.cc:2258] T b8e0070d176c4b69ad2aca72bbb7c102 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:45.279949 13164 tablet_service.cc:1514] Processing DeleteTablet for tablet b8e0070d176c4b69ad2aca72bbb7c102 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:45 UTC) from {username='slave'} at 127.0.0.1:41668
I20240503 13:56:45.280025 21707 raft_consensus.cc:2229] T b8e0070d176c4b69ad2aca72bbb7c102 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:56:45.280225 21707 raft_consensus.cc:2258] T b8e0070d176c4b69ad2aca72bbb7c102 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:45.280584 20120 raft_consensus.cc:504] T abc88caa63f34af48f92e81918cf1d4e P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:45.280689 21706 ts_tablet_manager.cc:1875] T b8e0070d176c4b69ad2aca72bbb7c102 P 009f849ee7b54c80a3f5264695047e4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:45.280912 20120 leader_election.cc:288] T abc88caa63f34af48f92e81918cf1d4e P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 election: Requested vote from peers 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885), 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153)
I20240503 13:56:45.281232 21707 ts_tablet_manager.cc:1875] T b8e0070d176c4b69ad2aca72bbb7c102 P 76c386bef61a4bd5a907266a5118ffe2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:45.281376 15758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abc88caa63f34af48f92e81918cf1d4e" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2"
I20240503 13:56:45.281471 15758 raft_consensus.cc:3037] T abc88caa63f34af48f92e81918cf1d4e P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:45.282333 15758 raft_consensus.cc:2454] T abc88caa63f34af48f92e81918cf1d4e P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 1.
I20240503 13:56:45.282696 15493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abc88caa63f34af48f92e81918cf1d4e" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546"
I20240503 13:56:45.282814 15493 raft_consensus.cc:3037] T abc88caa63f34af48f92e81918cf1d4e P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:45.283283 21707 ts_tablet_manager.cc:1888] T b8e0070d176c4b69ad2aca72bbb7c102 P 76c386bef61a4bd5a907266a5118ffe2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:45.283345 21707 log.cc:1196] T b8e0070d176c4b69ad2aca72bbb7c102 P 76c386bef61a4bd5a907266a5118ffe2: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/wal/wals/b8e0070d176c4b69ad2aca72bbb7c102
I20240503 13:56:45.283336 21706 ts_tablet_manager.cc:1888] T b8e0070d176c4b69ad2aca72bbb7c102 P 009f849ee7b54c80a3f5264695047e4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:45.283408 21706 log.cc:1196] T b8e0070d176c4b69ad2aca72bbb7c102 P 009f849ee7b54c80a3f5264695047e4d: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/wal/wals/b8e0070d176c4b69ad2aca72bbb7c102
I20240503 13:56:45.283622 21707 ts_tablet_manager.cc:1909] T b8e0070d176c4b69ad2aca72bbb7c102 P 76c386bef61a4bd5a907266a5118ffe2: Deleting consensus metadata
I20240503 13:56:45.283658 21706 ts_tablet_manager.cc:1909] T b8e0070d176c4b69ad2aca72bbb7c102 P 009f849ee7b54c80a3f5264695047e4d: Deleting consensus metadata
I20240503 13:56:45.283815 15493 raft_consensus.cc:2454] T abc88caa63f34af48f92e81918cf1d4e P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 1.
I20240503 13:56:45.283859 21708 tablet_replica.cc:330] T b8e0070d176c4b69ad2aca72bbb7c102 P 8ccf5a4ac450414c9d3579bd5800b770: stopping tablet replica
I20240503 13:56:45.283996 21708 raft_consensus.cc:2229] T b8e0070d176c4b69ad2aca72bbb7c102 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:45.284301 12896 catalog_manager.cc:4830] TS 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951): tablet b8e0070d176c4b69ad2aca72bbb7c102 (table rebalancer_dynamic_table_new_name [id=c82e4bf13b5f46c6a7475628085ae2cf]) successfully deleted
I20240503 13:56:45.284334 13383 leader_election.cc:302] T abc88caa63f34af48f92e81918cf1d4e P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61b887f0d40d497badeca5604c1f0546, 7690ee79577c45e3a2c8f8b552c0bff8; no voters:
I20240503 13:56:45.284519 20120 raft_consensus.cc:2781] T abc88caa63f34af48f92e81918cf1d4e P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:45.284736 20120 raft_consensus.cc:686] T abc88caa63f34af48f92e81918cf1d4e P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 LEADER]: Becoming Leader. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Running, Role: LEADER
I20240503 13:56:45.284739 12899 catalog_manager.cc:4830] TS 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885): tablet b8e0070d176c4b69ad2aca72bbb7c102 (table rebalancer_dynamic_table_new_name [id=c82e4bf13b5f46c6a7475628085ae2cf]) successfully deleted
I20240503 13:56:45.284838 21708 raft_consensus.cc:2258] T b8e0070d176c4b69ad2aca72bbb7c102 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:45.284876 20120 consensus_queue.cc:237] T abc88caa63f34af48f92e81918cf1d4e P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:45.286201 21708 ts_tablet_manager.cc:1875] T b8e0070d176c4b69ad2aca72bbb7c102 P 8ccf5a4ac450414c9d3579bd5800b770: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:45.286449 15624 consensus_queue.cc:237] T 6bca3e19362842fe928adb84e7c9297a P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:45.286698 12904 catalog_manager.cc:5484] T abc88caa63f34af48f92e81918cf1d4e P 7690ee79577c45e3a2c8f8b552c0bff8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68). New cstate: current_term: 1 leader_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:45.287705 13447 raft_consensus.cc:1261] T 6bca3e19362842fe928adb84e7c9297a P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer 009f849ee7b54c80a3f5264695047e4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:56:45.288051 15758 raft_consensus.cc:1261] T 6bca3e19362842fe928adb84e7c9297a P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Refusing update from remote peer 009f849ee7b54c80a3f5264695047e4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:56:45.288440 21515 consensus_queue.cc:1035] T 6bca3e19362842fe928adb84e7c9297a P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:56:45.288769 21191 consensus_queue.cc:1035] T 6bca3e19362842fe928adb84e7c9297a P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:56:45.289044 21708 ts_tablet_manager.cc:1888] T b8e0070d176c4b69ad2aca72bbb7c102 P 8ccf5a4ac450414c9d3579bd5800b770: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:45.289112 21708 log.cc:1196] T b8e0070d176c4b69ad2aca72bbb7c102 P 8ccf5a4ac450414c9d3579bd5800b770: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/wal/wals/b8e0070d176c4b69ad2aca72bbb7c102
I20240503 13:56:45.289391 21708 ts_tablet_manager.cc:1909] T b8e0070d176c4b69ad2aca72bbb7c102 P 8ccf5a4ac450414c9d3579bd5800b770: Deleting consensus metadata
I20240503 13:56:45.290172 12899 catalog_manager.cc:4830] TS 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387): tablet b8e0070d176c4b69ad2aca72bbb7c102 (table rebalancer_dynamic_table_new_name [id=c82e4bf13b5f46c6a7475628085ae2cf]) successfully deleted
I20240503 13:56:45.290715 21515 raft_consensus.cc:2932] T 6bca3e19362842fe928adb84e7c9297a P 009f849ee7b54c80a3f5264695047e4d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } }
I20240503 13:56:45.291323 13447 raft_consensus.cc:2932] T 6bca3e19362842fe928adb84e7c9297a P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } }
I20240503 13:56:45.292693 15758 raft_consensus.cc:2932] T 6bca3e19362842fe928adb84e7c9297a P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } }
I20240503 13:56:45.295181 12904 catalog_manager.cc:5484] T 6bca3e19362842fe928adb84e7c9297a P 009f849ee7b54c80a3f5264695047e4d reported cstate change: config changed from index 2 to 3, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) evicted. New cstate: current_term: 1 leader_uuid: "009f849ee7b54c80a3f5264695047e4d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } }
I20240503 13:56:45.298960 15868 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bca3e19362842fe928adb84e7c9297a with delete_type TABLET_DATA_TOMBSTONED (TS e47105a0f46e4511b7112e571688a4c7 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:50498
I20240503 13:56:45.299309 21720 tablet_replica.cc:330] T 6bca3e19362842fe928adb84e7c9297a P e47105a0f46e4511b7112e571688a4c7: stopping tablet replica
I20240503 13:56:45.299475 21720 raft_consensus.cc:2229] T 6bca3e19362842fe928adb84e7c9297a P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:45.299712 21720 raft_consensus.cc:2258] T 6bca3e19362842fe928adb84e7c9297a P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:45.300194 21720 ts_tablet_manager.cc:1875] T 6bca3e19362842fe928adb84e7c9297a P e47105a0f46e4511b7112e571688a4c7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:56:45.302548 21720 ts_tablet_manager.cc:1888] T 6bca3e19362842fe928adb84e7c9297a P e47105a0f46e4511b7112e571688a4c7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 13:56:45.302616 21720 log.cc:1196] T 6bca3e19362842fe928adb84e7c9297a P e47105a0f46e4511b7112e571688a4c7: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-7/wal/wals/6bca3e19362842fe928adb84e7c9297a
I20240503 13:56:45.303138 12896 catalog_manager.cc:4830] TS e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): tablet 6bca3e19362842fe928adb84e7c9297a (table rebalancer_extra_table_4 [id=51ad326dfca444208bdd2a00d9582876]) successfully deleted
I20240503 13:56:45.380151 12906 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:56:45.387501 13296 tablet_service.cc:1467] Processing CreateTablet for tablet e027368334844943a3cbb4c34187bcbf (DEFAULT_TABLE table=rebalancer_dynamic_table [id=11162e0e34094e618d335d322a673326]), partition=
I20240503 13:56:45.387630 13164 tablet_service.cc:1467] Processing CreateTablet for tablet e027368334844943a3cbb4c34187bcbf (DEFAULT_TABLE table=rebalancer_dynamic_table [id=11162e0e34094e618d335d322a673326]), partition=
I20240503 13:56:45.387804 13296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e027368334844943a3cbb4c34187bcbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:45.387809 13424 tablet_service.cc:1467] Processing CreateTablet for tablet e027368334844943a3cbb4c34187bcbf (DEFAULT_TABLE table=rebalancer_dynamic_table [id=11162e0e34094e618d335d322a673326]), partition=
I20240503 13:56:45.387889 13164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e027368334844943a3cbb4c34187bcbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:45.388048 13424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e027368334844943a3cbb4c34187bcbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:45.390128 21684 tablet_bootstrap.cc:492] T e027368334844943a3cbb4c34187bcbf P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:45.391008 21684 tablet_bootstrap.cc:654] T e027368334844943a3cbb4c34187bcbf P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:45.394194 21515 raft_consensus.cc:482] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:45.394299 21515 raft_consensus.cc:504] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:45.394690 21515 leader_election.cc:288] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:45.395051 13051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8438207c32194339a43e203f751d2345" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e" is_pre_election: true
I20240503 13:56:45.395185 13051 raft_consensus.cc:2454] T 8438207c32194339a43e203f751d2345 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 0.
I20240503 13:56:45.395422 21684 tablet_bootstrap.cc:492] T e027368334844943a3cbb4c34187bcbf P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:45.395529 21684 ts_tablet_manager.cc:1367] T e027368334844943a3cbb4c34187bcbf P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:56:45.395825 15888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8438207c32194339a43e203f751d2345" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7" is_pre_election: true
I20240503 13:56:45.395856 15559 leader_election.cc:302] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 60ba3ffe3b7746469834cd91c84be14e; no voters:
I20240503 13:56:45.395953 15888 raft_consensus.cc:2454] T 8438207c32194339a43e203f751d2345 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 0.
I20240503 13:56:45.396039 21684 raft_consensus.cc:348] T e027368334844943a3cbb4c34187bcbf P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:45.396129 21684 raft_consensus.cc:374] T e027368334844943a3cbb4c34187bcbf P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:45.396163 21684 raft_consensus.cc:729] T e027368334844943a3cbb4c34187bcbf P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:45.396288 21684 consensus_queue.cc:260] T e027368334844943a3cbb4c34187bcbf P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:45.396696 21684 ts_tablet_manager.cc:1398] T e027368334844943a3cbb4c34187bcbf P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:45.396911 21515 raft_consensus.cc:2781] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:45.396982 21515 raft_consensus.cc:482] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:45.397024 21515 raft_consensus.cc:3037] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:45.398046 21515 raft_consensus.cc:504] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:45.398398 21515 leader_election.cc:288] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 election: Requested vote from peers 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:45.399034 21757 tablet_bootstrap.cc:492] T e027368334844943a3cbb4c34187bcbf P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:45.399055 13051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8438207c32194339a43e203f751d2345" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e"
I20240503 13:56:45.399120 15888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8438207c32194339a43e203f751d2345" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7"
I20240503 13:56:45.399180 13051 raft_consensus.cc:3037] T 8438207c32194339a43e203f751d2345 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:45.399224 15888 raft_consensus.cc:3037] T 8438207c32194339a43e203f751d2345 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:45.400015 21757 tablet_bootstrap.cc:654] T e027368334844943a3cbb4c34187bcbf P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:45.400139 13051 raft_consensus.cc:2454] T 8438207c32194339a43e203f751d2345 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 1.
I20240503 13:56:45.400272 15888 raft_consensus.cc:2454] T 8438207c32194339a43e203f751d2345 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 1.
I20240503 13:56:45.400568 15559 leader_election.cc:302] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 60ba3ffe3b7746469834cd91c84be14e; no voters:
I20240503 13:56:45.400753 21515 raft_consensus.cc:2781] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:45.400839 21515 raft_consensus.cc:686] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d [term 1 LEADER]: Becoming Leader. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Running, Role: LEADER
I20240503 13:56:45.401001 21515 consensus_queue.cc:237] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:45.403154 12908 catalog_manager.cc:5484] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d reported cstate change: term changed from 0 to 1, leader changed from <none> to 009f849ee7b54c80a3f5264695047e4d (127.6.98.70). New cstate: current_term: 1 leader_uuid: "009f849ee7b54c80a3f5264695047e4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:45.404364 21757 tablet_bootstrap.cc:492] T e027368334844943a3cbb4c34187bcbf P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:45.404541 21757 ts_tablet_manager.cc:1367] T e027368334844943a3cbb4c34187bcbf P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 13:56:45.405086 21757 raft_consensus.cc:348] T e027368334844943a3cbb4c34187bcbf P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:45.405305 21757 raft_consensus.cc:374] T e027368334844943a3cbb4c34187bcbf P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:45.405386 21757 raft_consensus.cc:729] T e027368334844943a3cbb4c34187bcbf P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:45.405581 21757 consensus_queue.cc:260] T e027368334844943a3cbb4c34187bcbf P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:45.406538 21757 ts_tablet_manager.cc:1398] T e027368334844943a3cbb4c34187bcbf P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:45.408705 21758 tablet_bootstrap.cc:492] T e027368334844943a3cbb4c34187bcbf P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:45.409690 21758 tablet_bootstrap.cc:654] T e027368334844943a3cbb4c34187bcbf P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:45.410777 21758 tablet_bootstrap.cc:492] T e027368334844943a3cbb4c34187bcbf P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:45.410861 21758 ts_tablet_manager.cc:1367] T e027368334844943a3cbb4c34187bcbf P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:56:45.411244 21758 raft_consensus.cc:348] T e027368334844943a3cbb4c34187bcbf P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:45.411337 21758 raft_consensus.cc:374] T e027368334844943a3cbb4c34187bcbf P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:45.411397 21758 raft_consensus.cc:729] T e027368334844943a3cbb4c34187bcbf P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:45.411517 21758 consensus_queue.cc:260] T e027368334844943a3cbb4c34187bcbf P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:45.411756 21758 ts_tablet_manager.cc:1398] T e027368334844943a3cbb4c34187bcbf P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:45.435885 21191 raft_consensus.cc:482] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:45.436000 21191 raft_consensus.cc:504] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:45.436389 21191 leader_election.cc:288] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:45.436820 15888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dd22ebd569c4d1da07bcb3365973406" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7" is_pre_election: true
I20240503 13:56:45.436821 15493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dd22ebd569c4d1da07bcb3365973406" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546" is_pre_election: true
I20240503 13:56:45.436965 15888 raft_consensus.cc:2454] T 9dd22ebd569c4d1da07bcb3365973406 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 0.
I20240503 13:56:45.436991 15493 raft_consensus.cc:2454] T 9dd22ebd569c4d1da07bcb3365973406 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 0.
I20240503 13:56:45.437213 15558 leader_election.cc:302] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:45.437376 21191 raft_consensus.cc:2781] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:45.437439 21191 raft_consensus.cc:482] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:45.437466 21191 raft_consensus.cc:3037] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:45.438730 21191 raft_consensus.cc:504] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:45.439060 21191 leader_election.cc:288] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 election: Requested vote from peers 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:45.439368 15493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dd22ebd569c4d1da07bcb3365973406" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546"
I20240503 13:56:45.439383 15888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dd22ebd569c4d1da07bcb3365973406" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7"
I20240503 13:56:45.439498 15888 raft_consensus.cc:3037] T 9dd22ebd569c4d1da07bcb3365973406 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:45.439505 15493 raft_consensus.cc:3037] T 9dd22ebd569c4d1da07bcb3365973406 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:45.440444 15493 raft_consensus.cc:2454] T 9dd22ebd569c4d1da07bcb3365973406 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 1.
I20240503 13:56:45.440444 15888 raft_consensus.cc:2454] T 9dd22ebd569c4d1da07bcb3365973406 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 1.
I20240503 13:56:45.440721 15558 leader_election.cc:302] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:45.440888 21191 raft_consensus.cc:2781] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:45.440971 21191 raft_consensus.cc:686] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d [term 1 LEADER]: Becoming Leader. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Running, Role: LEADER
I20240503 13:56:45.441275 21191 consensus_queue.cc:237] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:45.443145 12908 catalog_manager.cc:5484] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d reported cstate change: term changed from 0 to 1, leader changed from <none> to 009f849ee7b54c80a3f5264695047e4d (127.6.98.70). New cstate: current_term: 1 leader_uuid: "009f849ee7b54c80a3f5264695047e4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:45.458979 15605 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 13:56:45.459975 20120 raft_consensus.cc:482] T e027368334844943a3cbb4c34187bcbf P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:45.460074 20120 raft_consensus.cc:504] T e027368334844943a3cbb4c34187bcbf P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:45.460423 20120 leader_election.cc:288] T e027368334844943a3cbb4c34187bcbf P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:45.460753 15471 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 13:56:45.461041 13185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e027368334844943a3cbb4c34187bcbf" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770" is_pre_election: true
I20240503 13:56:45.461189 13185 raft_consensus.cc:2454] T e027368334844943a3cbb4c34187bcbf P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 0.
I20240503 13:56:45.461346 13314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e027368334844943a3cbb4c34187bcbf" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703" is_pre_election: true
I20240503 13:56:45.461474 13314 raft_consensus.cc:2454] T e027368334844943a3cbb4c34187bcbf P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 0.
I20240503 13:56:45.462738 13383 leader_election.cc:302] T e027368334844943a3cbb4c34187bcbf P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, 8ccf5a4ac450414c9d3579bd5800b770; no voters:
I20240503 13:56:45.463107 21709 raft_consensus.cc:2781] T e027368334844943a3cbb4c34187bcbf P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:45.463165 21709 raft_consensus.cc:482] T e027368334844943a3cbb4c34187bcbf P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:45.463213 21709 raft_consensus.cc:3037] T e027368334844943a3cbb4c34187bcbf P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:45.464413 21709 raft_consensus.cc:504] T e027368334844943a3cbb4c34187bcbf P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:45.464802 21709 leader_election.cc:288] T e027368334844943a3cbb4c34187bcbf P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 election: Requested vote from peers ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:45.467099 13032 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:45.468006 13314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e027368334844943a3cbb4c34187bcbf" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703"
I20240503 13:56:45.468129 13314 raft_consensus.cc:3037] T e027368334844943a3cbb4c34187bcbf P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:45.468097 13185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e027368334844943a3cbb4c34187bcbf" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770"
I20240503 13:56:45.468241 13185 raft_consensus.cc:3037] T e027368334844943a3cbb4c34187bcbf P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:45.469180 13314 raft_consensus.cc:2454] T e027368334844943a3cbb4c34187bcbf P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 1.
I20240503 13:56:45.469239 13185 raft_consensus.cc:2454] T e027368334844943a3cbb4c34187bcbf P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 1.
I20240503 13:56:45.470947 13383 leader_election.cc:302] T e027368334844943a3cbb4c34187bcbf P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, 8ccf5a4ac450414c9d3579bd5800b770; no voters:
I20240503 13:56:45.471104 21709 raft_consensus.cc:2781] T e027368334844943a3cbb4c34187bcbf P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:45.471186 21709 raft_consensus.cc:686] T e027368334844943a3cbb4c34187bcbf P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 LEADER]: Becoming Leader. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Running, Role: LEADER
I20240503 13:56:45.471314 21709 consensus_queue.cc:237] T e027368334844943a3cbb4c34187bcbf P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:45.474241 12906 catalog_manager.cc:5484] T e027368334844943a3cbb4c34187bcbf P 7690ee79577c45e3a2c8f8b552c0bff8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68). New cstate: current_term: 1 leader_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } }
I20240503 13:56:45.476518 15738 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 13:56:45.477739 15868 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:56:45.482141 13296 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 13:56:45.490166 13164 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 13:56:45.508940 13424 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 13:56:45.560864 13447 consensus_queue.cc:237] T 356ccd6fdf25470dba2a79367a37128e P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:45.562165 13185 raft_consensus.cc:1261] T 356ccd6fdf25470dba2a79367a37128e P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Refusing update from remote peer 7690ee79577c45e3a2c8f8b552c0bff8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:56:45.562330 15758 raft_consensus.cc:1261] T 356ccd6fdf25470dba2a79367a37128e P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Refusing update from remote peer 7690ee79577c45e3a2c8f8b552c0bff8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:56:45.562908 21709 consensus_queue.cc:1035] T 356ccd6fdf25470dba2a79367a37128e P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:56:45.563077 20120 consensus_queue.cc:1035] T 356ccd6fdf25470dba2a79367a37128e P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:56:45.564976 20120 raft_consensus.cc:2932] T 356ccd6fdf25470dba2a79367a37128e P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 61b887f0d40d497badeca5604c1f0546 (127.6.98.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } }
I20240503 13:56:45.565348 15758 raft_consensus.cc:2932] T 356ccd6fdf25470dba2a79367a37128e P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 61b887f0d40d497badeca5604c1f0546 (127.6.98.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } }
I20240503 13:56:45.565955 13185 raft_consensus.cc:2932] T 356ccd6fdf25470dba2a79367a37128e P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 61b887f0d40d497badeca5604c1f0546 (127.6.98.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } }
I20240503 13:56:45.568946 12908 catalog_manager.cc:5484] T 356ccd6fdf25470dba2a79367a37128e P 7690ee79577c45e3a2c8f8b552c0bff8 reported cstate change: config changed from index 2 to 3, VOTER 61b887f0d40d497badeca5604c1f0546 (127.6.98.69) evicted. New cstate: current_term: 1 leader_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } }
I20240503 13:56:45.573392 15471 tablet_service.cc:1514] Processing DeleteTablet for tablet 356ccd6fdf25470dba2a79367a37128e with delete_type TABLET_DATA_TOMBSTONED (TS 61b887f0d40d497badeca5604c1f0546 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:58758
I20240503 13:56:45.574775 21799 tablet_replica.cc:330] T 356ccd6fdf25470dba2a79367a37128e P 61b887f0d40d497badeca5604c1f0546: stopping tablet replica
I20240503 13:56:45.574944 21799 raft_consensus.cc:2229] T 356ccd6fdf25470dba2a79367a37128e P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:45.575088 21799 raft_consensus.cc:2258] T 356ccd6fdf25470dba2a79367a37128e P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:45.575489 21799 ts_tablet_manager.cc:1875] T 356ccd6fdf25470dba2a79367a37128e P 61b887f0d40d497badeca5604c1f0546: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:56:45.577653 21799 ts_tablet_manager.cc:1888] T 356ccd6fdf25470dba2a79367a37128e P 61b887f0d40d497badeca5604c1f0546: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 13:56:45.577733 21799 log.cc:1196] T 356ccd6fdf25470dba2a79367a37128e P 61b887f0d40d497badeca5604c1f0546: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/wal/wals/356ccd6fdf25470dba2a79367a37128e
I20240503 13:56:45.578248 12899 catalog_manager.cc:4830] TS 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153): tablet 356ccd6fdf25470dba2a79367a37128e (table rebalancer_extra_table_0 [id=18c0e326c0a144c382cd95a8c0052057]) successfully deleted
I20240503 13:56:45.580054 12912 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:56:45.584924 13185 raft_consensus.cc:1261] T e027368334844943a3cbb4c34187bcbf P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Refusing update from remote peer 7690ee79577c45e3a2c8f8b552c0bff8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:45.585026 13314 raft_consensus.cc:1261] T e027368334844943a3cbb4c34187bcbf P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Refusing update from remote peer 7690ee79577c45e3a2c8f8b552c0bff8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:45.585429 21709 consensus_queue.cc:1035] T e027368334844943a3cbb4c34187bcbf P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:45.585606 20120 consensus_queue.cc:1035] T e027368334844943a3cbb4c34187bcbf P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:45.589313 12906 catalog_manager.cc:5576] TS 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387) does not have the latest schema for tablet e027368334844943a3cbb4c34187bcbf (table rebalancer_dynamic_table [id=11162e0e34094e618d335d322a673326]). Expected version 1 got 0
I20240503 13:56:45.589727 21435 tablet.cc:1722] T e027368334844943a3cbb4c34187bcbf P 8ccf5a4ac450414c9d3579bd5800b770: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:45.590063 21435 tablet.cc:1620] T e027368334844943a3cbb4c34187bcbf P 8ccf5a4ac450414c9d3579bd5800b770: MemRowSet was empty: no flush needed.
I20240503 13:56:45.590312 12908 catalog_manager.cc:5576] TS ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279) does not have the latest schema for tablet e027368334844943a3cbb4c34187bcbf (table rebalancer_dynamic_table [id=11162e0e34094e618d335d322a673326]). Expected version 1 got 0
I20240503 13:56:45.590795 21810 tablet.cc:1722] T e027368334844943a3cbb4c34187bcbf P 7690ee79577c45e3a2c8f8b552c0bff8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:45.594264 21811 tablet.cc:1722] T e027368334844943a3cbb4c34187bcbf P ba0194098aa84449ae17e8a0d4c43703: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:45.594822 12908 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=11162e0e34094e618d335d322a673326] alter complete (version 1)
I20240503 13:56:45.595464 21811 tablet.cc:1620] T e027368334844943a3cbb4c34187bcbf P ba0194098aa84449ae17e8a0d4c43703: MemRowSet was empty: no flush needed.
I20240503 13:56:45.600762 21810 tablet.cc:1620] T e027368334844943a3cbb4c34187bcbf P 7690ee79577c45e3a2c8f8b552c0bff8: MemRowSet was empty: no flush needed.
I20240503 13:56:45.684643 13032 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:45.685164 15471 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 13:56:45.694168 13164 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 13:56:45.695298 20120 consensus_queue.cc:1035] T abc88caa63f34af48f92e81918cf1d4e P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:45.699607 15605 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 13:56:45.700707 12909 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:56:45.708112 21709 consensus_queue.cc:1035] T abc88caa63f34af48f92e81918cf1d4e P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:45.712368 21810 tablet.cc:1722] T e027368334844943a3cbb4c34187bcbf P 7690ee79577c45e3a2c8f8b552c0bff8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:45.712740 21810 tablet.cc:1620] T e027368334844943a3cbb4c34187bcbf P 7690ee79577c45e3a2c8f8b552c0bff8: MemRowSet was empty: no flush needed.
I20240503 13:56:45.713359 15738 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 13:56:45.719936 21811 tablet.cc:1722] T e027368334844943a3cbb4c34187bcbf P ba0194098aa84449ae17e8a0d4c43703: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:45.720306 21811 tablet.cc:1620] T e027368334844943a3cbb4c34187bcbf P ba0194098aa84449ae17e8a0d4c43703: MemRowSet was empty: no flush needed.
I20240503 13:56:45.720675 12909 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=11162e0e34094e618d335d322a673326] alter complete (version 2)
I20240503 13:56:45.721644 13296 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 13:56:45.726007 21435 tablet.cc:1722] T e027368334844943a3cbb4c34187bcbf P 8ccf5a4ac450414c9d3579bd5800b770: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:45.726337 21435 tablet.cc:1620] T e027368334844943a3cbb4c34187bcbf P 8ccf5a4ac450414c9d3579bd5800b770: MemRowSet was empty: no flush needed.
I20240503 13:56:45.730406 13424 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 13:56:45.734337 15868 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:56:45.801218 13314 consensus_queue.cc:237] T 258d1ec1f0694c15a4d41bd064f35b4e P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:45.802610 15493 raft_consensus.cc:1261] T 258d1ec1f0694c15a4d41bd064f35b4e P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:56:45.802634 13185 raft_consensus.cc:1261] T 258d1ec1f0694c15a4d41bd064f35b4e P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:56:45.802968 20215 consensus_queue.cc:1035] T 258d1ec1f0694c15a4d41bd064f35b4e P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:45.803170 21234 consensus_queue.cc:1035] T 258d1ec1f0694c15a4d41bd064f35b4e P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:45.805133 21234 raft_consensus.cc:2932] T 258d1ec1f0694c15a4d41bd064f35b4e P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:45.806890 15493 raft_consensus.cc:2932] T 258d1ec1f0694c15a4d41bd064f35b4e P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:45.807401 13185 raft_consensus.cc:2932] T 258d1ec1f0694c15a4d41bd064f35b4e P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:45.807914 12909 catalog_manager.cc:5484] T 258d1ec1f0694c15a4d41bd064f35b4e P ba0194098aa84449ae17e8a0d4c43703 reported cstate change: config changed from index 4 to 5, VOTER 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68) evicted. New cstate: current_term: 1 leader_uuid: "ba0194098aa84449ae17e8a0d4c43703" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:45.811048 13424 tablet_service.cc:1514] Processing DeleteTablet for tablet 258d1ec1f0694c15a4d41bd064f35b4e with delete_type TABLET_DATA_TOMBSTONED (TS 7690ee79577c45e3a2c8f8b552c0bff8 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:33678
I20240503 13:56:45.811493 21876 tablet_replica.cc:330] T 258d1ec1f0694c15a4d41bd064f35b4e P 7690ee79577c45e3a2c8f8b552c0bff8: stopping tablet replica
I20240503 13:56:45.811810 21876 raft_consensus.cc:2229] T 258d1ec1f0694c15a4d41bd064f35b4e P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:45.811993 21876 raft_consensus.cc:2258] T 258d1ec1f0694c15a4d41bd064f35b4e P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:45.812695 21876 ts_tablet_manager.cc:1875] T 258d1ec1f0694c15a4d41bd064f35b4e P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:56:45.815340 21876 ts_tablet_manager.cc:1888] T 258d1ec1f0694c15a4d41bd064f35b4e P 7690ee79577c45e3a2c8f8b552c0bff8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 13:56:45.815464 21876 log.cc:1196] T 258d1ec1f0694c15a4d41bd064f35b4e P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal/wals/258d1ec1f0694c15a4d41bd064f35b4e
I20240503 13:56:45.816823 12896 catalog_manager.cc:4830] TS 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109): tablet 258d1ec1f0694c15a4d41bd064f35b4e (table rebalance_test_table_7 [id=5e6f2f3703e1424ab61d8fe49e6266dd]) successfully deleted
I20240503 13:56:45.825850 12912 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:56:45.831337 21810 tablet.cc:1722] T e027368334844943a3cbb4c34187bcbf P 7690ee79577c45e3a2c8f8b552c0bff8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:45.831794 21811 tablet.cc:1722] T e027368334844943a3cbb4c34187bcbf P ba0194098aa84449ae17e8a0d4c43703: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:45.832207 21435 tablet.cc:1722] T e027368334844943a3cbb4c34187bcbf P 8ccf5a4ac450414c9d3579bd5800b770: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:45.837455 12912 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table_new_name [id=11162e0e34094e618d335d322a673326] alter complete (version 3)
I20240503 13:56:45.875463 21515 consensus_queue.cc:1035] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:45.880379 21191 consensus_queue.cc:1035] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:45.907879 21191 consensus_queue.cc:1035] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:45.911931 21191 consensus_queue.cc:1035] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:45.931859 15605 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 13:56:45.932153 13424 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 13:56:45.933450 15471 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 13:56:45.936913 13032 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:45.943177 15738 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 13:56:45.947223 12906 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:45.947383 12906 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:45.949725 12906 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Sending DeleteTablet for 3 replicas of tablet e027368334844943a3cbb4c34187bcbf
I20240503 13:56:45.950526 13424 tablet_service.cc:1514] Processing DeleteTablet for tablet e027368334844943a3cbb4c34187bcbf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:45 UTC) from {username='slave'} at 127.0.0.1:33678
I20240503 13:56:45.950978 21876 tablet_replica.cc:330] T e027368334844943a3cbb4c34187bcbf P 7690ee79577c45e3a2c8f8b552c0bff8: stopping tablet replica
I20240503 13:56:45.951143 21876 raft_consensus.cc:2229] T e027368334844943a3cbb4c34187bcbf P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:56:45.951342 21876 raft_consensus.cc:2258] T e027368334844943a3cbb4c34187bcbf P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:45.950534 13164 tablet_service.cc:1514] Processing DeleteTablet for tablet e027368334844943a3cbb4c34187bcbf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:45 UTC) from {username='slave'} at 127.0.0.1:41668
I20240503 13:56:45.954274 21876 ts_tablet_manager.cc:1875] T e027368334844943a3cbb4c34187bcbf P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:45.961136 13296 tablet_service.cc:1514] Processing DeleteTablet for tablet e027368334844943a3cbb4c34187bcbf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:45 UTC) from {username='slave'} at 127.0.0.1:33250
I20240503 13:56:45.961827 21937 tablet_replica.cc:330] T e027368334844943a3cbb4c34187bcbf P 8ccf5a4ac450414c9d3579bd5800b770: stopping tablet replica
I20240503 13:56:45.962051 21937 raft_consensus.cc:2229] T e027368334844943a3cbb4c34187bcbf P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:45.963404 21937 raft_consensus.cc:2258] T e027368334844943a3cbb4c34187bcbf P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:45.964092 21876 ts_tablet_manager.cc:1888] T e027368334844943a3cbb4c34187bcbf P 7690ee79577c45e3a2c8f8b552c0bff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:45.964181 21876 log.cc:1196] T e027368334844943a3cbb4c34187bcbf P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal/wals/e027368334844943a3cbb4c34187bcbf
I20240503 13:56:45.964802 21937 ts_tablet_manager.cc:1875] T e027368334844943a3cbb4c34187bcbf P 8ccf5a4ac450414c9d3579bd5800b770: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:45.967881 21937 ts_tablet_manager.cc:1888] T e027368334844943a3cbb4c34187bcbf P 8ccf5a4ac450414c9d3579bd5800b770: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:45.968017 21937 log.cc:1196] T e027368334844943a3cbb4c34187bcbf P 8ccf5a4ac450414c9d3579bd5800b770: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/wal/wals/e027368334844943a3cbb4c34187bcbf
I20240503 13:56:45.968343 13164 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 13:56:45.964469 21876 ts_tablet_manager.cc:1909] T e027368334844943a3cbb4c34187bcbf P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting consensus metadata
I20240503 13:56:45.968827 21937 ts_tablet_manager.cc:1909] T e027368334844943a3cbb4c34187bcbf P 8ccf5a4ac450414c9d3579bd5800b770: Deleting consensus metadata
I20240503 13:56:45.969394 12896 catalog_manager.cc:4830] TS 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109): tablet e027368334844943a3cbb4c34187bcbf (table rebalancer_dynamic_table_new_name [id=11162e0e34094e618d335d322a673326]) successfully deleted
I20240503 13:56:45.969614 12899 catalog_manager.cc:4830] TS 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387): tablet e027368334844943a3cbb4c34187bcbf (table rebalancer_dynamic_table_new_name [id=11162e0e34094e618d335d322a673326]) successfully deleted
I20240503 13:56:45.970350 12906 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_extra_table_44"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:56:45.976550 15605 tablet_service.cc:1467] Processing CreateTablet for tablet b9da0bf8a4814a93aa3d066440d01a88 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=5550e602537c4f66907d0d27f3d04306]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:45.976794 15604 tablet_service.cc:1467] Processing CreateTablet for tablet dc91cd8f4c0d48ae922fd4d5f12d161f (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=5550e602537c4f66907d0d27f3d04306]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:45.976864 15605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9da0bf8a4814a93aa3d066440d01a88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:45.977051 15604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc91cd8f4c0d48ae922fd4d5f12d161f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:45.977656 15471 tablet_service.cc:1467] Processing CreateTablet for tablet b9da0bf8a4814a93aa3d066440d01a88 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=5550e602537c4f66907d0d27f3d04306]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:45.977936 15471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9da0bf8a4814a93aa3d066440d01a88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:45.979048 21939 tablet_bootstrap.cc:492] T dc91cd8f4c0d48ae922fd4d5f12d161f P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:45.979209 15473 tablet_service.cc:1467] Processing CreateTablet for tablet 2095ba0a3e264d79bdc7f3dd3fc6a3b5 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=5550e602537c4f66907d0d27f3d04306]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:45.979420 15473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2095ba0a3e264d79bdc7f3dd3fc6a3b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:45.979885 13164 tablet_service.cc:1467] Processing CreateTablet for tablet b9da0bf8a4814a93aa3d066440d01a88 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=5550e602537c4f66907d0d27f3d04306]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:45.980139 13164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9da0bf8a4814a93aa3d066440d01a88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:45.981134 13424 tablet_service.cc:1467] Processing CreateTablet for tablet dc91cd8f4c0d48ae922fd4d5f12d161f (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=5550e602537c4f66907d0d27f3d04306]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:45.979216 15472 tablet_service.cc:1467] Processing CreateTablet for tablet dc91cd8f4c0d48ae922fd4d5f12d161f (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=5550e602537c4f66907d0d27f3d04306]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:45.981387 15472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc91cd8f4c0d48ae922fd4d5f12d161f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:45.981397 13424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc91cd8f4c0d48ae922fd4d5f12d161f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:45.981513 15868 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:56:45.981591 15867 tablet_service.cc:1467] Processing CreateTablet for tablet 2095ba0a3e264d79bdc7f3dd3fc6a3b5 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=5550e602537c4f66907d0d27f3d04306]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:45.981822 15867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2095ba0a3e264d79bdc7f3dd3fc6a3b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:45.983345 15605 tablet_service.cc:1467] Processing CreateTablet for tablet 2095ba0a3e264d79bdc7f3dd3fc6a3b5 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=5550e602537c4f66907d0d27f3d04306]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:45.983573 15605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2095ba0a3e264d79bdc7f3dd3fc6a3b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:45.985395 21939 tablet_bootstrap.cc:654] T dc91cd8f4c0d48ae922fd4d5f12d161f P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:45.986135 21941 tablet_bootstrap.cc:492] T dc91cd8f4c0d48ae922fd4d5f12d161f P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:45.987140 21941 tablet_bootstrap.cc:654] T dc91cd8f4c0d48ae922fd4d5f12d161f P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:45.988260 21941 tablet_bootstrap.cc:492] T dc91cd8f4c0d48ae922fd4d5f12d161f P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:45.988343 21941 ts_tablet_manager.cc:1367] T dc91cd8f4c0d48ae922fd4d5f12d161f P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:56:45.988816 21941 raft_consensus.cc:348] T dc91cd8f4c0d48ae922fd4d5f12d161f P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:45.988916 21941 raft_consensus.cc:374] T dc91cd8f4c0d48ae922fd4d5f12d161f P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:45.988951 21941 raft_consensus.cc:729] T dc91cd8f4c0d48ae922fd4d5f12d161f P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:45.989063 21941 consensus_queue.cc:260] T dc91cd8f4c0d48ae922fd4d5f12d161f P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:45.989300 21941 ts_tablet_manager.cc:1398] T dc91cd8f4c0d48ae922fd4d5f12d161f P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:45.996351 21942 tablet_bootstrap.cc:492] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:45.997139 21940 tablet_bootstrap.cc:492] T b9da0bf8a4814a93aa3d066440d01a88 P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:45.997459 21942 tablet_bootstrap.cc:654] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P e47105a0f46e4511b7112e571688a4c7: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:45.998340 21940 tablet_bootstrap.cc:654] T b9da0bf8a4814a93aa3d066440d01a88 P 61b887f0d40d497badeca5604c1f0546: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:46.001245 21945 tablet_bootstrap.cc:492] T b9da0bf8a4814a93aa3d066440d01a88 P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:46.002276 21945 tablet_bootstrap.cc:654] T b9da0bf8a4814a93aa3d066440d01a88 P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:46.003459 21942 tablet_bootstrap.cc:492] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P e47105a0f46e4511b7112e571688a4c7: No bootstrap required, opened a new log
I20240503 13:56:46.003556 21942 ts_tablet_manager.cc:1367] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 13:56:46.003868 21938 tablet_replica.cc:330] T e027368334844943a3cbb4c34187bcbf P ba0194098aa84449ae17e8a0d4c43703: stopping tablet replica
I20240503 13:56:46.004029 21938 raft_consensus.cc:2229] T e027368334844943a3cbb4c34187bcbf P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:46.003990 21942 raft_consensus.cc:348] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:46.004096 21942 raft_consensus.cc:374] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:46.004122 21942 raft_consensus.cc:729] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:46.004141 21938 raft_consensus.cc:2258] T e027368334844943a3cbb4c34187bcbf P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:46.004240 21942 consensus_queue.cc:260] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:46.004530 21942 ts_tablet_manager.cc:1398] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:46.006742 21938 ts_tablet_manager.cc:1875] T e027368334844943a3cbb4c34187bcbf P ba0194098aa84449ae17e8a0d4c43703: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:46.007198 21940 tablet_bootstrap.cc:492] T b9da0bf8a4814a93aa3d066440d01a88 P 61b887f0d40d497badeca5604c1f0546: No bootstrap required, opened a new log
I20240503 13:56:46.007354 21940 ts_tablet_manager.cc:1367] T b9da0bf8a4814a93aa3d066440d01a88 P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 13:56:46.007956 21940 raft_consensus.cc:348] T b9da0bf8a4814a93aa3d066440d01a88 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:46.008325 21940 raft_consensus.cc:374] T b9da0bf8a4814a93aa3d066440d01a88 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:46.008419 21940 raft_consensus.cc:729] T b9da0bf8a4814a93aa3d066440d01a88 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:46.008679 21940 consensus_queue.cc:260] T b9da0bf8a4814a93aa3d066440d01a88 P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:46.008294 13296 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 13:56:46.009822 21940 ts_tablet_manager.cc:1398] T b9da0bf8a4814a93aa3d066440d01a88 P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:46.010037 21940 tablet_bootstrap.cc:492] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:46.010977 21940 tablet_bootstrap.cc:654] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 61b887f0d40d497badeca5604c1f0546: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:46.011862 21938 ts_tablet_manager.cc:1888] T e027368334844943a3cbb4c34187bcbf P ba0194098aa84449ae17e8a0d4c43703: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:46.011932 21938 log.cc:1196] T e027368334844943a3cbb4c34187bcbf P ba0194098aa84449ae17e8a0d4c43703: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/wal/wals/e027368334844943a3cbb4c34187bcbf
I20240503 13:56:46.013312 21945 tablet_bootstrap.cc:492] T b9da0bf8a4814a93aa3d066440d01a88 P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:46.013429 21945 ts_tablet_manager.cc:1367] T b9da0bf8a4814a93aa3d066440d01a88 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 13:56:46.013904 21945 raft_consensus.cc:348] T b9da0bf8a4814a93aa3d066440d01a88 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:46.014029 21945 raft_consensus.cc:374] T b9da0bf8a4814a93aa3d066440d01a88 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:46.014060 21945 raft_consensus.cc:729] T b9da0bf8a4814a93aa3d066440d01a88 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:46.014070 21939 tablet_bootstrap.cc:492] T dc91cd8f4c0d48ae922fd4d5f12d161f P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:46.014158 21939 ts_tablet_manager.cc:1367] T dc91cd8f4c0d48ae922fd4d5f12d161f P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.035s user 0.002s sys 0.000s
I20240503 13:56:46.014174 21945 consensus_queue.cc:260] T b9da0bf8a4814a93aa3d066440d01a88 P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:46.014480 21945 ts_tablet_manager.cc:1398] T b9da0bf8a4814a93aa3d066440d01a88 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:46.014572 21939 raft_consensus.cc:348] T dc91cd8f4c0d48ae922fd4d5f12d161f P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:46.014672 21939 raft_consensus.cc:374] T dc91cd8f4c0d48ae922fd4d5f12d161f P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:46.014698 21939 raft_consensus.cc:729] T dc91cd8f4c0d48ae922fd4d5f12d161f P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:46.014822 21939 consensus_queue.cc:260] T dc91cd8f4c0d48ae922fd4d5f12d161f P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:46.015079 21939 ts_tablet_manager.cc:1398] T dc91cd8f4c0d48ae922fd4d5f12d161f P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:46.015226 21939 tablet_bootstrap.cc:492] T b9da0bf8a4814a93aa3d066440d01a88 P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
W20240503 13:56:46.015738 13494 tablet.cc:2369] T dc91cd8f4c0d48ae922fd4d5f12d161f P 7690ee79577c45e3a2c8f8b552c0bff8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 13:56:46.016155 21939 tablet_bootstrap.cc:654] T b9da0bf8a4814a93aa3d066440d01a88 P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:46.017441 21939 tablet_bootstrap.cc:492] T b9da0bf8a4814a93aa3d066440d01a88 P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:46.017524 21939 ts_tablet_manager.cc:1367] T b9da0bf8a4814a93aa3d066440d01a88 P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:56:46.018234 21939 raft_consensus.cc:348] T b9da0bf8a4814a93aa3d066440d01a88 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:46.019457 21938 ts_tablet_manager.cc:1909] T e027368334844943a3cbb4c34187bcbf P ba0194098aa84449ae17e8a0d4c43703: Deleting consensus metadata
I20240503 13:56:46.019467 21939 raft_consensus.cc:374] T b9da0bf8a4814a93aa3d066440d01a88 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:46.019519 21939 raft_consensus.cc:729] T b9da0bf8a4814a93aa3d066440d01a88 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:46.019697 21939 consensus_queue.cc:260] T b9da0bf8a4814a93aa3d066440d01a88 P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:46.020015 21939 ts_tablet_manager.cc:1398] T b9da0bf8a4814a93aa3d066440d01a88 P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:46.020220 12899 catalog_manager.cc:4830] TS ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279): tablet e027368334844943a3cbb4c34187bcbf (table rebalancer_dynamic_table_new_name [id=11162e0e34094e618d335d322a673326]) successfully deleted
I20240503 13:56:46.020967 21939 tablet_bootstrap.cc:492] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:46.022027 21939 tablet_bootstrap.cc:654] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:46.024981 21939 tablet_bootstrap.cc:492] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:46.025072 21939 ts_tablet_manager.cc:1367] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:56:46.025511 21939 raft_consensus.cc:348] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:46.025606 21939 raft_consensus.cc:374] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:46.025633 21939 raft_consensus.cc:729] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:46.025739 21939 consensus_queue.cc:260] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:46.025873 21515 raft_consensus.cc:482] T b9da0bf8a4814a93aa3d066440d01a88 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:46.025972 21939 ts_tablet_manager.cc:1398] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:46.025947 21515 raft_consensus.cc:504] T b9da0bf8a4814a93aa3d066440d01a88 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:46.026273 21515 leader_election.cc:288] T b9da0bf8a4814a93aa3d066440d01a88 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387), 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153)
I20240503 13:56:46.026592 13185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9da0bf8a4814a93aa3d066440d01a88" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770" is_pre_election: true
I20240503 13:56:46.026711 13185 raft_consensus.cc:2454] T b9da0bf8a4814a93aa3d066440d01a88 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 0.
I20240503 13:56:46.026980 15561 leader_election.cc:302] T b9da0bf8a4814a93aa3d066440d01a88 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 8ccf5a4ac450414c9d3579bd5800b770; no voters:
I20240503 13:56:46.027127 21515 raft_consensus.cc:2781] T b9da0bf8a4814a93aa3d066440d01a88 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:46.027182 21515 raft_consensus.cc:482] T b9da0bf8a4814a93aa3d066440d01a88 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:46.027223 21515 raft_consensus.cc:3037] T b9da0bf8a4814a93aa3d066440d01a88 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:46.028144 21515 raft_consensus.cc:504] T b9da0bf8a4814a93aa3d066440d01a88 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:46.028529 21515 leader_election.cc:288] T b9da0bf8a4814a93aa3d066440d01a88 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 election: Requested vote from peers 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387), 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153)
I20240503 13:56:46.028781 13185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9da0bf8a4814a93aa3d066440d01a88" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770"
I20240503 13:56:46.028887 13185 raft_consensus.cc:3037] T b9da0bf8a4814a93aa3d066440d01a88 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:46.029057 15492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9da0bf8a4814a93aa3d066440d01a88" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546"
I20240503 13:56:46.029184 15492 raft_consensus.cc:3037] T b9da0bf8a4814a93aa3d066440d01a88 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:46.029050 15493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9da0bf8a4814a93aa3d066440d01a88" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546" is_pre_election: true
I20240503 13:56:46.029932 13185 raft_consensus.cc:2454] T b9da0bf8a4814a93aa3d066440d01a88 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 1.
I20240503 13:56:46.030131 15492 raft_consensus.cc:2454] T b9da0bf8a4814a93aa3d066440d01a88 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 1.
I20240503 13:56:46.030320 15561 leader_election.cc:302] T b9da0bf8a4814a93aa3d066440d01a88 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 8ccf5a4ac450414c9d3579bd5800b770; no voters:
I20240503 13:56:46.030557 21191 raft_consensus.cc:2781] T b9da0bf8a4814a93aa3d066440d01a88 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:46.030637 21191 raft_consensus.cc:686] T b9da0bf8a4814a93aa3d066440d01a88 P 009f849ee7b54c80a3f5264695047e4d [term 1 LEADER]: Becoming Leader. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Running, Role: LEADER
I20240503 13:56:46.030755 21191 consensus_queue.cc:237] T b9da0bf8a4814a93aa3d066440d01a88 P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:46.033011 12911 catalog_manager.cc:5484] T b9da0bf8a4814a93aa3d066440d01a88 P 009f849ee7b54c80a3f5264695047e4d reported cstate change: term changed from 0 to 1, leader changed from <none> to 009f849ee7b54c80a3f5264695047e4d (127.6.98.70). New cstate: current_term: 1 leader_uuid: "009f849ee7b54c80a3f5264695047e4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:46.052052 12911 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:56:46.057638 13032 tablet_service.cc:1467] Processing CreateTablet for tablet ecb9718fc43d41d0a718db196c3e01ac (DEFAULT_TABLE table=rebalancer_dynamic_table [id=46acc8415e42486ea37f81bdaa8680b0]), partition=
I20240503 13:56:46.057832 15738 tablet_service.cc:1467] Processing CreateTablet for tablet ecb9718fc43d41d0a718db196c3e01ac (DEFAULT_TABLE table=rebalancer_dynamic_table [id=46acc8415e42486ea37f81bdaa8680b0]), partition=
I20240503 13:56:46.057920 13032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecb9718fc43d41d0a718db196c3e01ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:46.058037 15738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecb9718fc43d41d0a718db196c3e01ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:46.060142 21958 tablet_bootstrap.cc:492] T ecb9718fc43d41d0a718db196c3e01ac P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:46.060359 21959 tablet_bootstrap.cc:492] T ecb9718fc43d41d0a718db196c3e01ac P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:46.060447 13424 tablet_service.cc:1467] Processing CreateTablet for tablet ecb9718fc43d41d0a718db196c3e01ac (DEFAULT_TABLE table=rebalancer_dynamic_table [id=46acc8415e42486ea37f81bdaa8680b0]), partition=
I20240503 13:56:46.060719 13424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecb9718fc43d41d0a718db196c3e01ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:46.060882 21940 tablet_bootstrap.cc:492] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 61b887f0d40d497badeca5604c1f0546: No bootstrap required, opened a new log
I20240503 13:56:46.060976 21940 ts_tablet_manager.cc:1367] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.051s user 0.002s sys 0.000s
I20240503 13:56:46.061462 21940 raft_consensus.cc:348] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:46.061578 21940 raft_consensus.cc:374] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:46.061609 21940 raft_consensus.cc:729] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:46.061733 21940 consensus_queue.cc:260] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:46.062055 21940 ts_tablet_manager.cc:1398] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:46.062130 21958 tablet_bootstrap.cc:654] T ecb9718fc43d41d0a718db196c3e01ac P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:46.062229 21940 tablet_bootstrap.cc:492] T dc91cd8f4c0d48ae922fd4d5f12d161f P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:46.062955 21959 tablet_bootstrap.cc:654] T ecb9718fc43d41d0a718db196c3e01ac P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:46.063102 21940 tablet_bootstrap.cc:654] T dc91cd8f4c0d48ae922fd4d5f12d161f P 61b887f0d40d497badeca5604c1f0546: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:46.063989 21959 tablet_bootstrap.cc:492] T ecb9718fc43d41d0a718db196c3e01ac P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:46.064006 21958 tablet_bootstrap.cc:492] T ecb9718fc43d41d0a718db196c3e01ac P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:46.064078 21959 ts_tablet_manager.cc:1367] T ecb9718fc43d41d0a718db196c3e01ac P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:56:46.064082 21958 ts_tablet_manager.cc:1367] T ecb9718fc43d41d0a718db196c3e01ac P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:56:46.064553 21959 raft_consensus.cc:348] T ecb9718fc43d41d0a718db196c3e01ac P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:46.064640 21941 tablet_bootstrap.cc:492] T ecb9718fc43d41d0a718db196c3e01ac P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:46.064766 21959 raft_consensus.cc:374] T ecb9718fc43d41d0a718db196c3e01ac P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:46.064857 21959 raft_consensus.cc:729] T ecb9718fc43d41d0a718db196c3e01ac P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:46.064970 21958 raft_consensus.cc:348] T ecb9718fc43d41d0a718db196c3e01ac P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:46.065024 21959 consensus_queue.cc:260] T ecb9718fc43d41d0a718db196c3e01ac P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:46.065152 21958 raft_consensus.cc:374] T ecb9718fc43d41d0a718db196c3e01ac P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:46.065196 21958 raft_consensus.cc:729] T ecb9718fc43d41d0a718db196c3e01ac P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:46.065322 21958 consensus_queue.cc:260] T ecb9718fc43d41d0a718db196c3e01ac P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:46.065521 21941 tablet_bootstrap.cc:654] T ecb9718fc43d41d0a718db196c3e01ac P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:46.065611 21959 ts_tablet_manager.cc:1398] T ecb9718fc43d41d0a718db196c3e01ac P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:46.066452 21958 ts_tablet_manager.cc:1398] T ecb9718fc43d41d0a718db196c3e01ac P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:46.067656 21940 tablet_bootstrap.cc:492] T dc91cd8f4c0d48ae922fd4d5f12d161f P 61b887f0d40d497badeca5604c1f0546: No bootstrap required, opened a new log
I20240503 13:56:46.067837 21940 ts_tablet_manager.cc:1367] T dc91cd8f4c0d48ae922fd4d5f12d161f P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 13:56:46.068653 21940 raft_consensus.cc:348] T dc91cd8f4c0d48ae922fd4d5f12d161f P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:46.068826 21940 raft_consensus.cc:374] T dc91cd8f4c0d48ae922fd4d5f12d161f P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:46.068960 21940 raft_consensus.cc:729] T dc91cd8f4c0d48ae922fd4d5f12d161f P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:46.069140 21940 consensus_queue.cc:260] T dc91cd8f4c0d48ae922fd4d5f12d161f P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:46.070185 21940 ts_tablet_manager.cc:1398] T dc91cd8f4c0d48ae922fd4d5f12d161f P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:46.071614 21941 tablet_bootstrap.cc:492] T ecb9718fc43d41d0a718db196c3e01ac P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:46.071702 21941 ts_tablet_manager.cc:1367] T ecb9718fc43d41d0a718db196c3e01ac P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 13:56:46.072168 21941 raft_consensus.cc:348] T ecb9718fc43d41d0a718db196c3e01ac P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:46.072268 21941 raft_consensus.cc:374] T ecb9718fc43d41d0a718db196c3e01ac P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:46.072297 21941 raft_consensus.cc:729] T ecb9718fc43d41d0a718db196c3e01ac P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:46.072407 21941 consensus_queue.cc:260] T ecb9718fc43d41d0a718db196c3e01ac P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:46.072702 21941 ts_tablet_manager.cc:1398] T ecb9718fc43d41d0a718db196c3e01ac P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:46.081902 21515 raft_consensus.cc:482] T dc91cd8f4c0d48ae922fd4d5f12d161f P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:46.082015 21515 raft_consensus.cc:504] T dc91cd8f4c0d48ae922fd4d5f12d161f P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:46.082455 21515 leader_election.cc:288] T dc91cd8f4c0d48ae922fd4d5f12d161f P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153), 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:46.082839 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc91cd8f4c0d48ae922fd4d5f12d161f" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
I20240503 13:56:46.082901 15492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc91cd8f4c0d48ae922fd4d5f12d161f" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546" is_pre_election: true
I20240503 13:56:46.082966 13447 raft_consensus.cc:2454] T dc91cd8f4c0d48ae922fd4d5f12d161f P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 0.
I20240503 13:56:46.083001 15492 raft_consensus.cc:2454] T dc91cd8f4c0d48ae922fd4d5f12d161f P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 0.
I20240503 13:56:46.083258 15561 leader_election.cc:302] T dc91cd8f4c0d48ae922fd4d5f12d161f P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 61b887f0d40d497badeca5604c1f0546; no voters:
I20240503 13:56:46.083474 21515 raft_consensus.cc:2781] T dc91cd8f4c0d48ae922fd4d5f12d161f P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:46.083528 21515 raft_consensus.cc:482] T dc91cd8f4c0d48ae922fd4d5f12d161f P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:46.083563 21515 raft_consensus.cc:3037] T dc91cd8f4c0d48ae922fd4d5f12d161f P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:46.084591 21515 raft_consensus.cc:504] T dc91cd8f4c0d48ae922fd4d5f12d161f P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:46.084908 21515 leader_election.cc:288] T dc91cd8f4c0d48ae922fd4d5f12d161f P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 election: Requested vote from peers 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153), 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:46.085245 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc91cd8f4c0d48ae922fd4d5f12d161f" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
I20240503 13:56:46.085361 13447 raft_consensus.cc:3037] T dc91cd8f4c0d48ae922fd4d5f12d161f P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:46.085429 15492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc91cd8f4c0d48ae922fd4d5f12d161f" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546"
I20240503 13:56:46.085527 15492 raft_consensus.cc:3037] T dc91cd8f4c0d48ae922fd4d5f12d161f P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:46.086334 13447 raft_consensus.cc:2454] T dc91cd8f4c0d48ae922fd4d5f12d161f P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 1.
I20240503 13:56:46.086344 15492 raft_consensus.cc:2454] T dc91cd8f4c0d48ae922fd4d5f12d161f P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 1.
I20240503 13:56:46.086648 15558 leader_election.cc:302] T dc91cd8f4c0d48ae922fd4d5f12d161f P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 7690ee79577c45e3a2c8f8b552c0bff8; no voters:
I20240503 13:56:46.086795 21515 raft_consensus.cc:2781] T dc91cd8f4c0d48ae922fd4d5f12d161f P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:46.086869 21515 raft_consensus.cc:686] T dc91cd8f4c0d48ae922fd4d5f12d161f P 009f849ee7b54c80a3f5264695047e4d [term 1 LEADER]: Becoming Leader. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Running, Role: LEADER
I20240503 13:56:46.086994 21515 consensus_queue.cc:237] T dc91cd8f4c0d48ae922fd4d5f12d161f P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:46.087126 15624 consensus_queue.cc:237] T c8a53b7095e948f988c9151882bfa1bd P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:46.088441 13185 raft_consensus.cc:1261] T c8a53b7095e948f988c9151882bfa1bd P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Refusing update from remote peer 009f849ee7b54c80a3f5264695047e4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:56:46.089449 21975 consensus_queue.cc:1035] T c8a53b7095e948f988c9151882bfa1bd P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:56:46.089601 13314 raft_consensus.cc:1261] T c8a53b7095e948f988c9151882bfa1bd P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Refusing update from remote peer 009f849ee7b54c80a3f5264695047e4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:56:46.090082 21975 consensus_queue.cc:1035] T c8a53b7095e948f988c9151882bfa1bd P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:56:46.090940 12908 catalog_manager.cc:5484] T dc91cd8f4c0d48ae922fd4d5f12d161f P 009f849ee7b54c80a3f5264695047e4d reported cstate change: term changed from 0 to 1, leader changed from <none> to 009f849ee7b54c80a3f5264695047e4d (127.6.98.70). New cstate: current_term: 1 leader_uuid: "009f849ee7b54c80a3f5264695047e4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } }
I20240503 13:56:46.091895 21515 raft_consensus.cc:2932] T c8a53b7095e948f988c9151882bfa1bd P 009f849ee7b54c80a3f5264695047e4d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 61b887f0d40d497badeca5604c1f0546 (127.6.98.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } }
I20240503 13:56:46.092540 13185 raft_consensus.cc:2932] T c8a53b7095e948f988c9151882bfa1bd P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 61b887f0d40d497badeca5604c1f0546 (127.6.98.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } }
I20240503 13:56:46.093330 13314 raft_consensus.cc:2932] T c8a53b7095e948f988c9151882bfa1bd P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 61b887f0d40d497badeca5604c1f0546 (127.6.98.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } }
I20240503 13:56:46.095938 12911 catalog_manager.cc:5484] T c8a53b7095e948f988c9151882bfa1bd P ba0194098aa84449ae17e8a0d4c43703 reported cstate change: config changed from index 2 to 3, VOTER 61b887f0d40d497badeca5604c1f0546 (127.6.98.69) evicted. New cstate: current_term: 1 leader_uuid: "009f849ee7b54c80a3f5264695047e4d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } }
I20240503 13:56:46.098210 15472 tablet_service.cc:1514] Processing DeleteTablet for tablet c8a53b7095e948f988c9151882bfa1bd with delete_type TABLET_DATA_TOMBSTONED (TS 61b887f0d40d497badeca5604c1f0546 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:58758
I20240503 13:56:46.098543 21980 tablet_replica.cc:330] T c8a53b7095e948f988c9151882bfa1bd P 61b887f0d40d497badeca5604c1f0546: stopping tablet replica
I20240503 13:56:46.098691 21980 raft_consensus.cc:2229] T c8a53b7095e948f988c9151882bfa1bd P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:46.098805 21980 raft_consensus.cc:2258] T c8a53b7095e948f988c9151882bfa1bd P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:46.099213 21980 ts_tablet_manager.cc:1875] T c8a53b7095e948f988c9151882bfa1bd P 61b887f0d40d497badeca5604c1f0546: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:56:46.101399 21980 ts_tablet_manager.cc:1888] T c8a53b7095e948f988c9151882bfa1bd P 61b887f0d40d497badeca5604c1f0546: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 13:56:46.101517 21980 log.cc:1196] T c8a53b7095e948f988c9151882bfa1bd P 61b887f0d40d497badeca5604c1f0546: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/wal/wals/c8a53b7095e948f988c9151882bfa1bd
I20240503 13:56:46.102176 12899 catalog_manager.cc:4830] TS 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153): tablet c8a53b7095e948f988c9151882bfa1bd (table rebalancer_extra_table_8 [id=e28614beb29c450da9e4f209b2fbe1b2]) successfully deleted
W20240503 13:56:46.144102 13232 tablet.cc:2369] T b9da0bf8a4814a93aa3d066440d01a88 P 8ccf5a4ac450414c9d3579bd5800b770: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 13:56:46.195597 13101 tablet.cc:2369] T ecb9718fc43d41d0a718db196c3e01ac P 60ba3ffe3b7746469834cd91c84be14e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 13:56:46.321218 21975 raft_consensus.cc:482] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:46.321364 21975 raft_consensus.cc:504] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:46.321830 21975 leader_election.cc:288] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153)
I20240503 13:56:46.322139 15492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2095ba0a3e264d79bdc7f3dd3fc6a3b5" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546" is_pre_election: true
I20240503 13:56:46.322302 15492 raft_consensus.cc:2454] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 0.
I20240503 13:56:46.322297 15889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2095ba0a3e264d79bdc7f3dd3fc6a3b5" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7" is_pre_election: true
I20240503 13:56:46.322490 15889 raft_consensus.cc:2454] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 0.
I20240503 13:56:46.322875 15558 leader_election.cc:302] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:46.323205 21981 raft_consensus.cc:2781] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:46.323256 21981 raft_consensus.cc:482] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:46.323302 21981 raft_consensus.cc:3037] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:46.324404 21981 raft_consensus.cc:504] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:46.324808 21981 leader_election.cc:288] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 election: Requested vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153)
I20240503 13:56:46.325161 15492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2095ba0a3e264d79bdc7f3dd3fc6a3b5" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546"
I20240503 13:56:46.325439 15492 raft_consensus.cc:3037] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:46.325593 15888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2095ba0a3e264d79bdc7f3dd3fc6a3b5" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7"
I20240503 13:56:46.325704 15888 raft_consensus.cc:3037] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:46.326629 15492 raft_consensus.cc:2454] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 1.
I20240503 13:56:46.326742 15888 raft_consensus.cc:2454] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 1.
I20240503 13:56:46.327051 15561 leader_election.cc:302] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 61b887f0d40d497badeca5604c1f0546; no voters:
I20240503 13:56:46.327251 21981 raft_consensus.cc:2781] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:46.327369 21981 raft_consensus.cc:686] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d [term 1 LEADER]: Becoming Leader. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Running, Role: LEADER
I20240503 13:56:46.327497 21981 consensus_queue.cc:237] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:46.329725 12905 catalog_manager.cc:5484] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d reported cstate change: term changed from 0 to 1, leader changed from <none> to 009f849ee7b54c80a3f5264695047e4d (127.6.98.70). New cstate: current_term: 1 leader_uuid: "009f849ee7b54c80a3f5264695047e4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:46.332461 21252 raft_consensus.cc:482] T ecb9718fc43d41d0a718db196c3e01ac P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:46.332630 21252 raft_consensus.cc:504] T ecb9718fc43d41d0a718db196c3e01ac P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:46.333127 21252 leader_election.cc:288] T ecb9718fc43d41d0a718db196c3e01ac P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695), 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:46.333524 13052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecb9718fc43d41d0a718db196c3e01ac" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e" is_pre_election: true
I20240503 13:56:46.333556 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecb9718fc43d41d0a718db196c3e01ac" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
I20240503 13:56:46.333690 13052 raft_consensus.cc:2454] T ecb9718fc43d41d0a718db196c3e01ac P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 0.
I20240503 13:56:46.333709 13447 raft_consensus.cc:2454] T ecb9718fc43d41d0a718db196c3e01ac P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 0.
I20240503 13:56:46.334007 15692 leader_election.cc:302] T ecb9718fc43d41d0a718db196c3e01ac P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 7690ee79577c45e3a2c8f8b552c0bff8, 76c386bef61a4bd5a907266a5118ffe2; no voters:
I20240503 13:56:46.334204 21252 raft_consensus.cc:2781] T ecb9718fc43d41d0a718db196c3e01ac P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:46.334249 21252 raft_consensus.cc:482] T ecb9718fc43d41d0a718db196c3e01ac P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:46.334277 21252 raft_consensus.cc:3037] T ecb9718fc43d41d0a718db196c3e01ac P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:46.335274 21252 raft_consensus.cc:504] T ecb9718fc43d41d0a718db196c3e01ac P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:46.335563 21252 leader_election.cc:288] T ecb9718fc43d41d0a718db196c3e01ac P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 election: Requested vote from peers 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695), 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:46.335917 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecb9718fc43d41d0a718db196c3e01ac" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
I20240503 13:56:46.335930 13052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecb9718fc43d41d0a718db196c3e01ac" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e"
I20240503 13:56:46.336040 13052 raft_consensus.cc:3037] T ecb9718fc43d41d0a718db196c3e01ac P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:46.336040 13447 raft_consensus.cc:3037] T ecb9718fc43d41d0a718db196c3e01ac P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:46.337069 13447 raft_consensus.cc:2454] T ecb9718fc43d41d0a718db196c3e01ac P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 1.
I20240503 13:56:46.337069 13052 raft_consensus.cc:2454] T ecb9718fc43d41d0a718db196c3e01ac P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 1.
I20240503 13:56:46.337365 15691 leader_election.cc:302] T ecb9718fc43d41d0a718db196c3e01ac P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, 76c386bef61a4bd5a907266a5118ffe2; no voters:
I20240503 13:56:46.337538 21252 raft_consensus.cc:2781] T ecb9718fc43d41d0a718db196c3e01ac P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:46.337630 21252 raft_consensus.cc:686] T ecb9718fc43d41d0a718db196c3e01ac P 76c386bef61a4bd5a907266a5118ffe2 [term 1 LEADER]: Becoming Leader. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Running, Role: LEADER
I20240503 13:56:46.337759 21252 consensus_queue.cc:237] T ecb9718fc43d41d0a718db196c3e01ac P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:46.340126 12905 catalog_manager.cc:5484] T ecb9718fc43d41d0a718db196c3e01ac P 76c386bef61a4bd5a907266a5118ffe2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71). New cstate: current_term: 1 leader_uuid: "76c386bef61a4bd5a907266a5118ffe2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } }
I20240503 13:56:46.370802 13424 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:46.374053 15867 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:56:46.377660 13032 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:46.382321 15472 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 13:56:46.397365 13296 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 13:56:46.398370 15605 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:56:46.412076 13164 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 13:56:46.424170 15738 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 13:56:46.461833 15624 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3f5d536cffe742e69f4b0d3b93caf76c"
dest_uuid: "009f849ee7b54c80a3f5264695047e4d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38996
I20240503 13:56:46.462041 15624 raft_consensus.cc:595] T 3f5d536cffe742e69f4b0d3b93caf76c P 009f849ee7b54c80a3f5264695047e4d [term 1 LEADER]: Received request to transfer leadership
I20240503 13:56:46.465195 12905 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:56:46.470175 13445 raft_consensus.cc:1261] T ecb9718fc43d41d0a718db196c3e01ac P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer 76c386bef61a4bd5a907266a5118ffe2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:46.470666 19618 consensus_queue.cc:1035] T ecb9718fc43d41d0a718db196c3e01ac P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:46.472766 13052 raft_consensus.cc:1261] T ecb9718fc43d41d0a718db196c3e01ac P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Refusing update from remote peer 76c386bef61a4bd5a907266a5118ffe2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:46.473263 19618 consensus_queue.cc:1035] T ecb9718fc43d41d0a718db196c3e01ac P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:46.474072 22050 tablet.cc:1722] T ecb9718fc43d41d0a718db196c3e01ac P 76c386bef61a4bd5a907266a5118ffe2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:46.474329 12904 catalog_manager.cc:5576] TS 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109) does not have the latest schema for tablet ecb9718fc43d41d0a718db196c3e01ac (table rebalancer_dynamic_table [id=46acc8415e42486ea37f81bdaa8680b0]). Expected version 1 got 0
I20240503 13:56:46.474414 22050 tablet.cc:1620] T ecb9718fc43d41d0a718db196c3e01ac P 76c386bef61a4bd5a907266a5118ffe2: MemRowSet was empty: no flush needed.
I20240503 13:56:46.475234 22051 tablet.cc:1722] T ecb9718fc43d41d0a718db196c3e01ac P 7690ee79577c45e3a2c8f8b552c0bff8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:46.475531 22051 tablet.cc:1620] T ecb9718fc43d41d0a718db196c3e01ac P 7690ee79577c45e3a2c8f8b552c0bff8: MemRowSet was empty: no flush needed.
I20240503 13:56:46.479140 12905 catalog_manager.cc:5576] TS 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695) does not have the latest schema for tablet ecb9718fc43d41d0a718db196c3e01ac (table rebalancer_dynamic_table [id=46acc8415e42486ea37f81bdaa8680b0]). Expected version 1 got 0
I20240503 13:56:46.479322 12904 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=46acc8415e42486ea37f81bdaa8680b0] alter complete (version 1)
I20240503 13:56:46.482319 12910 catalog_manager.cc:5576] TS 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695) does not have the latest schema for tablet ecb9718fc43d41d0a718db196c3e01ac (table rebalancer_dynamic_table [id=46acc8415e42486ea37f81bdaa8680b0]). Expected version 1 got 0
I20240503 13:56:46.485179 22055 tablet.cc:1722] T ecb9718fc43d41d0a718db196c3e01ac P 60ba3ffe3b7746469834cd91c84be14e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:46.485494 22055 tablet.cc:1620] T ecb9718fc43d41d0a718db196c3e01ac P 60ba3ffe3b7746469834cd91c84be14e: MemRowSet was empty: no flush needed.
I20240503 13:56:46.576925 13032 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:46.579411 15472 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 13:56:46.580379 21981 consensus_queue.cc:1035] T b9da0bf8a4814a93aa3d066440d01a88 P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:46.582329 12907 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:56:46.591769 22050 tablet.cc:1722] T ecb9718fc43d41d0a718db196c3e01ac P 76c386bef61a4bd5a907266a5118ffe2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:46.592121 22050 tablet.cc:1620] T ecb9718fc43d41d0a718db196c3e01ac P 76c386bef61a4bd5a907266a5118ffe2: MemRowSet was empty: no flush needed.
I20240503 13:56:46.593152 22051 tablet.cc:1722] T ecb9718fc43d41d0a718db196c3e01ac P 7690ee79577c45e3a2c8f8b552c0bff8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:46.593454 22051 tablet.cc:1620] T ecb9718fc43d41d0a718db196c3e01ac P 7690ee79577c45e3a2c8f8b552c0bff8: MemRowSet was empty: no flush needed.
I20240503 13:56:46.595189 21979 consensus_queue.cc:1035] T dc91cd8f4c0d48ae922fd4d5f12d161f P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:46.598734 21979 raft_consensus.cc:979] T 3f5d536cffe742e69f4b0d3b93caf76c P 009f849ee7b54c80a3f5264695047e4d: : Instructing follower 60ba3ffe3b7746469834cd91c84be14e to start an election
I20240503 13:56:46.598839 21979 raft_consensus.cc:1067] T 3f5d536cffe742e69f4b0d3b93caf76c P 009f849ee7b54c80a3f5264695047e4d [term 1 LEADER]: Signalling peer 60ba3ffe3b7746469834cd91c84be14e to start an election
I20240503 13:56:46.599501 13052 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3f5d536cffe742e69f4b0d3b93caf76c"
dest_uuid: "60ba3ffe3b7746469834cd91c84be14e"
from {username='slave'} at 127.6.98.70:47015
I20240503 13:56:46.599642 13052 raft_consensus.cc:482] T 3f5d536cffe742e69f4b0d3b93caf76c P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 13:56:46.599692 13052 raft_consensus.cc:3037] T 3f5d536cffe742e69f4b0d3b93caf76c P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:56:46.599958 22055 tablet.cc:1722] T ecb9718fc43d41d0a718db196c3e01ac P 60ba3ffe3b7746469834cd91c84be14e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:46.600301 22055 tablet.cc:1620] T ecb9718fc43d41d0a718db196c3e01ac P 60ba3ffe3b7746469834cd91c84be14e: MemRowSet was empty: no flush needed.
I20240503 13:56:46.603760 13424 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:46.606572 21975 consensus_queue.cc:1035] T dc91cd8f4c0d48ae922fd4d5f12d161f P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:46.612800 12907 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=46acc8415e42486ea37f81bdaa8680b0] alter complete (version 2)
I20240503 13:56:46.615140 13052 raft_consensus.cc:504] T 3f5d536cffe742e69f4b0d3b93caf76c P 60ba3ffe3b7746469834cd91c84be14e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:46.615715 13052 leader_election.cc:288] T 3f5d536cffe742e69f4b0d3b93caf76c P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 2 election: Requested vote from peers 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885), 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951), ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:46.616124 13316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f5d536cffe742e69f4b0d3b93caf76c" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "ba0194098aa84449ae17e8a0d4c43703"
I20240503 13:56:46.616278 13316 raft_consensus.cc:3037] T 3f5d536cffe742e69f4b0d3b93caf76c P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:56:46.616344 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f5d536cffe742e69f4b0d3b93caf76c" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "76c386bef61a4bd5a907266a5118ffe2"
I20240503 13:56:46.616462 15757 raft_consensus.cc:3037] T 3f5d536cffe742e69f4b0d3b93caf76c P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:56:46.617492 13316 raft_consensus.cc:2454] T 3f5d536cffe742e69f4b0d3b93caf76c P ba0194098aa84449ae17e8a0d4c43703 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 2.
I20240503 13:56:46.617498 15757 raft_consensus.cc:2454] T 3f5d536cffe742e69f4b0d3b93caf76c P 76c386bef61a4bd5a907266a5118ffe2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 2.
I20240503 13:56:46.618149 12990 leader_election.cc:302] T 3f5d536cffe742e69f4b0d3b93caf76c P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 76c386bef61a4bd5a907266a5118ffe2, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:46.618278 15624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f5d536cffe742e69f4b0d3b93caf76c" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "009f849ee7b54c80a3f5264695047e4d"
I20240503 13:56:46.618367 22054 raft_consensus.cc:2781] T 3f5d536cffe742e69f4b0d3b93caf76c P 60ba3ffe3b7746469834cd91c84be14e [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:56:46.618400 15624 raft_consensus.cc:3032] T 3f5d536cffe742e69f4b0d3b93caf76c P 009f849ee7b54c80a3f5264695047e4d [term 1 LEADER]: Stepping down as leader of term 1
I20240503 13:56:46.618440 15624 raft_consensus.cc:729] T 3f5d536cffe742e69f4b0d3b93caf76c P 009f849ee7b54c80a3f5264695047e4d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Running, Role: LEADER
I20240503 13:56:46.618440 22054 raft_consensus.cc:686] T 3f5d536cffe742e69f4b0d3b93caf76c P 60ba3ffe3b7746469834cd91c84be14e [term 2 LEADER]: Becoming Leader. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Running, Role: LEADER
I20240503 13:56:46.618597 15624 consensus_queue.cc:260] T 3f5d536cffe742e69f4b0d3b93caf76c P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:46.618600 22054 consensus_queue.cc:237] T 3f5d536cffe742e69f4b0d3b93caf76c P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:46.618795 15624 raft_consensus.cc:3037] T 3f5d536cffe742e69f4b0d3b93caf76c P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:56:46.619386 15605 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 13:56:46.619886 15624 raft_consensus.cc:2454] T 3f5d536cffe742e69f4b0d3b93caf76c P 009f849ee7b54c80a3f5264695047e4d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 2.
I20240503 13:56:46.623764 12907 catalog_manager.cc:5484] T 3f5d536cffe742e69f4b0d3b93caf76c P 60ba3ffe3b7746469834cd91c84be14e reported cstate change: term changed from 1 to 2, leader changed from 009f849ee7b54c80a3f5264695047e4d (127.6.98.70) to 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65). New cstate: current_term: 2 leader_uuid: "60ba3ffe3b7746469834cd91c84be14e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:46.628832 13164 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 13:56:46.632570 15867 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:56:46.635762 21975 consensus_queue.cc:1035] T b9da0bf8a4814a93aa3d066440d01a88 P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:46.642849 15738 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 13:56:46.671072 13296 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 13:56:46.722465 12905 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:56:46.727838 21979 consensus_queue.cc:1035] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:46.728884 22050 tablet.cc:1722] T ecb9718fc43d41d0a718db196c3e01ac P 76c386bef61a4bd5a907266a5118ffe2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:46.730815 22051 tablet.cc:1722] T ecb9718fc43d41d0a718db196c3e01ac P 7690ee79577c45e3a2c8f8b552c0bff8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:46.734015 22055 tablet.cc:1722] T ecb9718fc43d41d0a718db196c3e01ac P 60ba3ffe3b7746469834cd91c84be14e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:46.740662 12912 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table_new_name [id=46acc8415e42486ea37f81bdaa8680b0] alter complete (version 3)
I20240503 13:56:46.750727 21975 consensus_queue.cc:1035] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:46.844270 12911 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:46.844431 12911 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:46.846107 13032 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 13:56:46.848141 15472 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 13:56:46.850488 12911 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Sending DeleteTablet for 3 replicas of tablet ecb9718fc43d41d0a718db196c3e01ac
I20240503 13:56:46.856894 13424 tablet_service.cc:1514] Processing DeleteTablet for tablet ecb9718fc43d41d0a718db196c3e01ac with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:46 UTC) from {username='slave'} at 127.0.0.1:33678
I20240503 13:56:46.857491 15738 tablet_service.cc:1514] Processing DeleteTablet for tablet ecb9718fc43d41d0a718db196c3e01ac with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:46 UTC) from {username='slave'} at 127.0.0.1:44984
I20240503 13:56:46.857518 13032 tablet_service.cc:1514] Processing DeleteTablet for tablet ecb9718fc43d41d0a718db196c3e01ac with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:46 UTC) from {username='slave'} at 127.0.0.1:59458
I20240503 13:56:46.857803 22179 tablet_replica.cc:330] T ecb9718fc43d41d0a718db196c3e01ac P 60ba3ffe3b7746469834cd91c84be14e: stopping tablet replica
I20240503 13:56:46.857805 13424 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:46.857951 22179 raft_consensus.cc:2229] T ecb9718fc43d41d0a718db196c3e01ac P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:46.858088 22179 raft_consensus.cc:2258] T ecb9718fc43d41d0a718db196c3e01ac P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:46.858721 15867 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:56:46.858981 22179 ts_tablet_manager.cc:1875] T ecb9718fc43d41d0a718db196c3e01ac P 60ba3ffe3b7746469834cd91c84be14e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:46.861282 22179 ts_tablet_manager.cc:1888] T ecb9718fc43d41d0a718db196c3e01ac P 60ba3ffe3b7746469834cd91c84be14e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:46.861423 22179 log.cc:1196] T ecb9718fc43d41d0a718db196c3e01ac P 60ba3ffe3b7746469834cd91c84be14e: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal/wals/ecb9718fc43d41d0a718db196c3e01ac
I20240503 13:56:46.861806 22179 ts_tablet_manager.cc:1909] T ecb9718fc43d41d0a718db196c3e01ac P 60ba3ffe3b7746469834cd91c84be14e: Deleting consensus metadata
I20240503 13:56:46.864852 12897 catalog_manager.cc:4830] TS 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695): tablet ecb9718fc43d41d0a718db196c3e01ac (table rebalancer_dynamic_table_new_name [id=46acc8415e42486ea37f81bdaa8680b0]) successfully deleted
I20240503 13:56:46.867578 13296 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 13:56:46.871181 15605 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 13:56:46.874580 22178 tablet_replica.cc:330] T ecb9718fc43d41d0a718db196c3e01ac P 76c386bef61a4bd5a907266a5118ffe2: stopping tablet replica
I20240503 13:56:46.874846 22178 raft_consensus.cc:2229] T ecb9718fc43d41d0a718db196c3e01ac P 76c386bef61a4bd5a907266a5118ffe2 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:56:46.875177 22178 raft_consensus.cc:2258] T ecb9718fc43d41d0a718db196c3e01ac P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:46.875830 22178 ts_tablet_manager.cc:1875] T ecb9718fc43d41d0a718db196c3e01ac P 76c386bef61a4bd5a907266a5118ffe2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:46.876047 22177 tablet_replica.cc:330] T ecb9718fc43d41d0a718db196c3e01ac P 7690ee79577c45e3a2c8f8b552c0bff8: stopping tablet replica
I20240503 13:56:46.876824 22177 raft_consensus.cc:2229] T ecb9718fc43d41d0a718db196c3e01ac P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:46.877121 22177 raft_consensus.cc:2258] T ecb9718fc43d41d0a718db196c3e01ac P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:46.878284 22178 ts_tablet_manager.cc:1888] T ecb9718fc43d41d0a718db196c3e01ac P 76c386bef61a4bd5a907266a5118ffe2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:46.878361 22178 log.cc:1196] T ecb9718fc43d41d0a718db196c3e01ac P 76c386bef61a4bd5a907266a5118ffe2: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/wal/wals/ecb9718fc43d41d0a718db196c3e01ac
I20240503 13:56:46.878652 22178 ts_tablet_manager.cc:1909] T ecb9718fc43d41d0a718db196c3e01ac P 76c386bef61a4bd5a907266a5118ffe2: Deleting consensus metadata
I20240503 13:56:46.881078 12899 catalog_manager.cc:4830] TS 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885): tablet ecb9718fc43d41d0a718db196c3e01ac (table rebalancer_dynamic_table_new_name [id=46acc8415e42486ea37f81bdaa8680b0]) successfully deleted
I20240503 13:56:46.881150 22177 ts_tablet_manager.cc:1875] T ecb9718fc43d41d0a718db196c3e01ac P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:46.882370 12911 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_extra_table_46"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:56:46.887706 22177 ts_tablet_manager.cc:1888] T ecb9718fc43d41d0a718db196c3e01ac P 7690ee79577c45e3a2c8f8b552c0bff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:46.887874 22177 log.cc:1196] T ecb9718fc43d41d0a718db196c3e01ac P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal/wals/ecb9718fc43d41d0a718db196c3e01ac
I20240503 13:56:46.888258 22177 ts_tablet_manager.cc:1909] T ecb9718fc43d41d0a718db196c3e01ac P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting consensus metadata
I20240503 13:56:46.888448 13164 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 13:56:46.888988 12896 catalog_manager.cc:4830] TS 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109): tablet ecb9718fc43d41d0a718db196c3e01ac (table rebalancer_dynamic_table_new_name [id=46acc8415e42486ea37f81bdaa8680b0]) successfully deleted
I20240503 13:56:46.890969 15738 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 13:56:46.892012 15605 tablet_service.cc:1467] Processing CreateTablet for tablet 1307b1fe211b4fd3947d1a7361cfd562 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=c2d6e174c9d04ff2a1f8bbaea85d2808]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:46.892314 15605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1307b1fe211b4fd3947d1a7361cfd562. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:46.892375 13032 tablet_service.cc:1467] Processing CreateTablet for tablet 4b7d0daf418b4516a7d9f70580d3d9c6 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=c2d6e174c9d04ff2a1f8bbaea85d2808]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:46.892654 13032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b7d0daf418b4516a7d9f70580d3d9c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:46.894675 22180 tablet_bootstrap.cc:492] T 4b7d0daf418b4516a7d9f70580d3d9c6 P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:46.894825 22181 tablet_bootstrap.cc:492] T 1307b1fe211b4fd3947d1a7361cfd562 P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:46.895080 13164 tablet_service.cc:1467] Processing CreateTablet for tablet 1307b1fe211b4fd3947d1a7361cfd562 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=c2d6e174c9d04ff2a1f8bbaea85d2808]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:46.895344 13164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1307b1fe211b4fd3947d1a7361cfd562. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:46.895754 22180 tablet_bootstrap.cc:654] T 4b7d0daf418b4516a7d9f70580d3d9c6 P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:46.895754 22181 tablet_bootstrap.cc:654] T 1307b1fe211b4fd3947d1a7361cfd562 P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:46.897339 22180 tablet_bootstrap.cc:492] T 4b7d0daf418b4516a7d9f70580d3d9c6 P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:46.897423 22180 ts_tablet_manager.cc:1367] T 4b7d0daf418b4516a7d9f70580d3d9c6 P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 13:56:46.897859 22180 raft_consensus.cc:348] T 4b7d0daf418b4516a7d9f70580d3d9c6 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:46.897966 22180 raft_consensus.cc:374] T 4b7d0daf418b4516a7d9f70580d3d9c6 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:46.897994 22180 raft_consensus.cc:729] T 4b7d0daf418b4516a7d9f70580d3d9c6 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:46.898131 22180 consensus_queue.cc:260] T 4b7d0daf418b4516a7d9f70580d3d9c6 P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:46.898396 22180 ts_tablet_manager.cc:1398] T 4b7d0daf418b4516a7d9f70580d3d9c6 P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:46.899197 13295 tablet_service.cc:1467] Processing CreateTablet for tablet 34188b6f8c654108aead589fe01f52d4 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=c2d6e174c9d04ff2a1f8bbaea85d2808]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:46.899487 13295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34188b6f8c654108aead589fe01f52d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:46.899909 15472 tablet_service.cc:1467] Processing CreateTablet for tablet 1307b1fe211b4fd3947d1a7361cfd562 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=c2d6e174c9d04ff2a1f8bbaea85d2808]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:46.900158 15472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1307b1fe211b4fd3947d1a7361cfd562. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:46.899197 13296 tablet_service.cc:1467] Processing CreateTablet for tablet 4b7d0daf418b4516a7d9f70580d3d9c6 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=c2d6e174c9d04ff2a1f8bbaea85d2808]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:46.902312 13296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b7d0daf418b4516a7d9f70580d3d9c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:46.902684 15473 tablet_service.cc:1467] Processing CreateTablet for tablet 34188b6f8c654108aead589fe01f52d4 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=c2d6e174c9d04ff2a1f8bbaea85d2808]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:46.902913 15473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34188b6f8c654108aead589fe01f52d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:46.903776 22184 tablet_bootstrap.cc:492] T 1307b1fe211b4fd3947d1a7361cfd562 P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:46.904865 22184 tablet_bootstrap.cc:654] T 1307b1fe211b4fd3947d1a7361cfd562 P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:46.905257 22186 tablet_bootstrap.cc:492] T 1307b1fe211b4fd3947d1a7361cfd562 P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:46.906137 22184 tablet_bootstrap.cc:492] T 1307b1fe211b4fd3947d1a7361cfd562 P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:46.906199 22186 tablet_bootstrap.cc:654] T 1307b1fe211b4fd3947d1a7361cfd562 P 61b887f0d40d497badeca5604c1f0546: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:46.906227 22184 ts_tablet_manager.cc:1367] T 1307b1fe211b4fd3947d1a7361cfd562 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:56:46.906694 22184 raft_consensus.cc:348] T 1307b1fe211b4fd3947d1a7361cfd562 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:46.906800 22184 raft_consensus.cc:374] T 1307b1fe211b4fd3947d1a7361cfd562 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:46.906828 22184 raft_consensus.cc:729] T 1307b1fe211b4fd3947d1a7361cfd562 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:46.906939 22184 consensus_queue.cc:260] T 1307b1fe211b4fd3947d1a7361cfd562 P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:46.907219 22184 ts_tablet_manager.cc:1398] T 1307b1fe211b4fd3947d1a7361cfd562 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:46.907387 22186 tablet_bootstrap.cc:492] T 1307b1fe211b4fd3947d1a7361cfd562 P 61b887f0d40d497badeca5604c1f0546: No bootstrap required, opened a new log
I20240503 13:56:46.907464 22186 ts_tablet_manager.cc:1367] T 1307b1fe211b4fd3947d1a7361cfd562 P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:56:46.907898 22186 raft_consensus.cc:348] T 1307b1fe211b4fd3947d1a7361cfd562 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:46.907999 22186 raft_consensus.cc:374] T 1307b1fe211b4fd3947d1a7361cfd562 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:46.908037 22186 raft_consensus.cc:729] T 1307b1fe211b4fd3947d1a7361cfd562 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:46.908145 22186 consensus_queue.cc:260] T 1307b1fe211b4fd3947d1a7361cfd562 P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:46.908389 22186 ts_tablet_manager.cc:1398] T 1307b1fe211b4fd3947d1a7361cfd562 P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:46.908563 22186 tablet_bootstrap.cc:492] T 34188b6f8c654108aead589fe01f52d4 P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:46.909456 22186 tablet_bootstrap.cc:654] T 34188b6f8c654108aead589fe01f52d4 P 61b887f0d40d497badeca5604c1f0546: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:46.911734 15738 tablet_service.cc:1467] Processing CreateTablet for tablet 4b7d0daf418b4516a7d9f70580d3d9c6 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=c2d6e174c9d04ff2a1f8bbaea85d2808]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:46.912052 15738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b7d0daf418b4516a7d9f70580d3d9c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:46.914608 13164 tablet_service.cc:1467] Processing CreateTablet for tablet 34188b6f8c654108aead589fe01f52d4 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=c2d6e174c9d04ff2a1f8bbaea85d2808]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:46.916463 22192 tablet_bootstrap.cc:492] T 4b7d0daf418b4516a7d9f70580d3d9c6 P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:46.917007 22186 tablet_bootstrap.cc:492] T 34188b6f8c654108aead589fe01f52d4 P 61b887f0d40d497badeca5604c1f0546: No bootstrap required, opened a new log
I20240503 13:56:46.917120 22186 ts_tablet_manager.cc:1367] T 34188b6f8c654108aead589fe01f52d4 P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240503 13:56:46.918135 22186 raft_consensus.cc:348] T 34188b6f8c654108aead589fe01f52d4 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:46.918241 22186 raft_consensus.cc:374] T 34188b6f8c654108aead589fe01f52d4 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:46.918268 22186 raft_consensus.cc:729] T 34188b6f8c654108aead589fe01f52d4 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:46.918383 22186 consensus_queue.cc:260] T 34188b6f8c654108aead589fe01f52d4 P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:46.918671 22186 ts_tablet_manager.cc:1398] T 34188b6f8c654108aead589fe01f52d4 P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:46.921180 22192 tablet_bootstrap.cc:654] T 4b7d0daf418b4516a7d9f70580d3d9c6 P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:46.921217 13164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34188b6f8c654108aead589fe01f52d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:46.922683 22192 tablet_bootstrap.cc:492] T 4b7d0daf418b4516a7d9f70580d3d9c6 P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:46.922950 22192 ts_tablet_manager.cc:1367] T 4b7d0daf418b4516a7d9f70580d3d9c6 P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20240503 13:56:46.923499 22192 raft_consensus.cc:348] T 4b7d0daf418b4516a7d9f70580d3d9c6 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:46.923686 22192 raft_consensus.cc:374] T 4b7d0daf418b4516a7d9f70580d3d9c6 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:46.923799 22192 raft_consensus.cc:729] T 4b7d0daf418b4516a7d9f70580d3d9c6 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:46.924063 22192 consensus_queue.cc:260] T 4b7d0daf418b4516a7d9f70580d3d9c6 P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:46.924427 22192 ts_tablet_manager.cc:1398] T 4b7d0daf418b4516a7d9f70580d3d9c6 P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:46.924765 22184 tablet_bootstrap.cc:492] T 34188b6f8c654108aead589fe01f52d4 P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:46.925822 22184 tablet_bootstrap.cc:654] T 34188b6f8c654108aead589fe01f52d4 P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:46.927389 22184 tablet_bootstrap.cc:492] T 34188b6f8c654108aead589fe01f52d4 P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:46.927484 22184 ts_tablet_manager.cc:1367] T 34188b6f8c654108aead589fe01f52d4 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:56:46.927934 22184 raft_consensus.cc:348] T 34188b6f8c654108aead589fe01f52d4 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:46.928051 22184 raft_consensus.cc:374] T 34188b6f8c654108aead589fe01f52d4 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:46.928078 22184 raft_consensus.cc:729] T 34188b6f8c654108aead589fe01f52d4 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:46.928185 22184 consensus_queue.cc:260] T 34188b6f8c654108aead589fe01f52d4 P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:46.928445 22184 ts_tablet_manager.cc:1398] T 34188b6f8c654108aead589fe01f52d4 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:46.952311 12903 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:56:46.953716 22185 tablet_bootstrap.cc:492] T 34188b6f8c654108aead589fe01f52d4 P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:46.955111 22185 tablet_bootstrap.cc:654] T 34188b6f8c654108aead589fe01f52d4 P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:46.957310 22181 tablet_bootstrap.cc:492] T 1307b1fe211b4fd3947d1a7361cfd562 P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:46.957415 22181 ts_tablet_manager.cc:1367] T 1307b1fe211b4fd3947d1a7361cfd562 P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.063s user 0.002s sys 0.000s
I20240503 13:56:46.957923 22181 raft_consensus.cc:348] T 1307b1fe211b4fd3947d1a7361cfd562 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:46.958038 22181 raft_consensus.cc:374] T 1307b1fe211b4fd3947d1a7361cfd562 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:46.958070 22181 raft_consensus.cc:729] T 1307b1fe211b4fd3947d1a7361cfd562 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:46.958185 22181 consensus_queue.cc:260] T 1307b1fe211b4fd3947d1a7361cfd562 P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:46.958464 22181 ts_tablet_manager.cc:1398] T 1307b1fe211b4fd3947d1a7361cfd562 P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:46.959708 15605 tablet_service.cc:1467] Processing CreateTablet for tablet a62d43de42a449d0af0c65117b53f041 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4fc2a17ebd4a4e1a8d55eb649fbf6da6]), partition=
I20240503 13:56:46.959843 15867 tablet_service.cc:1467] Processing CreateTablet for tablet a62d43de42a449d0af0c65117b53f041 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4fc2a17ebd4a4e1a8d55eb649fbf6da6]), partition=
I20240503 13:56:46.959949 15605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a62d43de42a449d0af0c65117b53f041. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:46.960068 15867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a62d43de42a449d0af0c65117b53f041. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:46.960716 13296 tablet_service.cc:1467] Processing CreateTablet for tablet a62d43de42a449d0af0c65117b53f041 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4fc2a17ebd4a4e1a8d55eb649fbf6da6]), partition=
I20240503 13:56:46.960930 13296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a62d43de42a449d0af0c65117b53f041. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:46.963344 22196 tablet_bootstrap.cc:492] T a62d43de42a449d0af0c65117b53f041 P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:46.964393 22196 tablet_bootstrap.cc:654] T a62d43de42a449d0af0c65117b53f041 P e47105a0f46e4511b7112e571688a4c7: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:46.966070 22181 tablet_bootstrap.cc:492] T a62d43de42a449d0af0c65117b53f041 P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:46.967041 22181 tablet_bootstrap.cc:654] T a62d43de42a449d0af0c65117b53f041 P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:46.968403 20722 raft_consensus.cc:482] T 34188b6f8c654108aead589fe01f52d4 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:46.968559 20722 raft_consensus.cc:504] T 34188b6f8c654108aead589fe01f52d4 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:46.968921 20722 leader_election.cc:288] T 34188b6f8c654108aead589fe01f52d4 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:46.969643 13185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34188b6f8c654108aead589fe01f52d4" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770" is_pre_election: true
I20240503 13:56:46.969776 13185 raft_consensus.cc:2454] T 34188b6f8c654108aead589fe01f52d4 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 0.
I20240503 13:56:46.970185 13316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34188b6f8c654108aead589fe01f52d4" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703" is_pre_election: true
I20240503 13:56:46.970657 15429 leader_election.cc:302] T 34188b6f8c654108aead589fe01f52d4 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61b887f0d40d497badeca5604c1f0546, 8ccf5a4ac450414c9d3579bd5800b770; no voters:
W20240503 13:56:46.970855 15429 leader_election.cc:341] T 34188b6f8c654108aead589fe01f52d4 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279): Illegal state: must be running to vote when last-logged opid is not known
I20240503 13:56:46.970992 20722 raft_consensus.cc:2781] T 34188b6f8c654108aead589fe01f52d4 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:46.971046 20722 raft_consensus.cc:482] T 34188b6f8c654108aead589fe01f52d4 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:46.971083 20722 raft_consensus.cc:3037] T 34188b6f8c654108aead589fe01f52d4 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:46.972065 20722 raft_consensus.cc:504] T 34188b6f8c654108aead589fe01f52d4 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:46.972384 20722 leader_election.cc:288] T 34188b6f8c654108aead589fe01f52d4 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 election: Requested vote from peers ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:46.972930 13316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34188b6f8c654108aead589fe01f52d4" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703"
I20240503 13:56:46.973425 22185 tablet_bootstrap.cc:492] T 34188b6f8c654108aead589fe01f52d4 P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:46.973467 22181 tablet_bootstrap.cc:492] T a62d43de42a449d0af0c65117b53f041 P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:46.973522 22185 ts_tablet_manager.cc:1367] T 34188b6f8c654108aead589fe01f52d4 P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20240503 13:56:46.973544 22181 ts_tablet_manager.cc:1367] T a62d43de42a449d0af0c65117b53f041 P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 13:56:46.973559 22196 tablet_bootstrap.cc:492] T a62d43de42a449d0af0c65117b53f041 P e47105a0f46e4511b7112e571688a4c7: No bootstrap required, opened a new log
I20240503 13:56:46.973644 22196 ts_tablet_manager.cc:1367] T a62d43de42a449d0af0c65117b53f041 P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 13:56:46.973975 22185 raft_consensus.cc:348] T 34188b6f8c654108aead589fe01f52d4 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:46.974071 22185 raft_consensus.cc:374] T 34188b6f8c654108aead589fe01f52d4 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:46.974099 22185 raft_consensus.cc:729] T 34188b6f8c654108aead589fe01f52d4 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:46.974047 22196 raft_consensus.cc:348] T a62d43de42a449d0af0c65117b53f041 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:46.974140 22196 raft_consensus.cc:374] T a62d43de42a449d0af0c65117b53f041 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:46.974114 22181 raft_consensus.cc:348] T a62d43de42a449d0af0c65117b53f041 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:46.974169 22196 raft_consensus.cc:729] T a62d43de42a449d0af0c65117b53f041 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:46.974198 22181 raft_consensus.cc:374] T a62d43de42a449d0af0c65117b53f041 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:46.974225 22181 raft_consensus.cc:729] T a62d43de42a449d0af0c65117b53f041 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:46.974223 22185 consensus_queue.cc:260] T 34188b6f8c654108aead589fe01f52d4 P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:46.974275 22196 consensus_queue.cc:260] T a62d43de42a449d0af0c65117b53f041 P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:46.974339 22181 consensus_queue.cc:260] T a62d43de42a449d0af0c65117b53f041 P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:46.974498 22185 ts_tablet_manager.cc:1398] T 34188b6f8c654108aead589fe01f52d4 P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:46.974524 22196 ts_tablet_manager.cc:1398] T a62d43de42a449d0af0c65117b53f041 P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:46.974628 22181 ts_tablet_manager.cc:1398] T a62d43de42a449d0af0c65117b53f041 P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:46.974676 22185 tablet_bootstrap.cc:492] T 4b7d0daf418b4516a7d9f70580d3d9c6 P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:46.974902 19618 raft_consensus.cc:482] T 4b7d0daf418b4516a7d9f70580d3d9c6 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:46.975014 19618 raft_consensus.cc:504] T 4b7d0daf418b4516a7d9f70580d3d9c6 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:46.975382 19618 leader_election.cc:288] T 4b7d0daf418b4516a7d9f70580d3d9c6 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279), 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:46.975466 22185 tablet_bootstrap.cc:654] T 4b7d0daf418b4516a7d9f70580d3d9c6 P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:46.976796 13051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b7d0daf418b4516a7d9f70580d3d9c6" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e" is_pre_election: true
I20240503 13:56:46.976922 13051 raft_consensus.cc:2454] T 4b7d0daf418b4516a7d9f70580d3d9c6 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 0.
I20240503 13:56:46.977562 22185 tablet_bootstrap.cc:492] T 4b7d0daf418b4516a7d9f70580d3d9c6 P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:46.978104 22185 ts_tablet_manager.cc:1367] T 4b7d0daf418b4516a7d9f70580d3d9c6 P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:56:46.978091 15692 leader_election.cc:302] T 4b7d0daf418b4516a7d9f70580d3d9c6 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 76c386bef61a4bd5a907266a5118ffe2; no voters:
I20240503 13:56:46.978502 22185 raft_consensus.cc:348] T 4b7d0daf418b4516a7d9f70580d3d9c6 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:46.978597 22185 raft_consensus.cc:374] T 4b7d0daf418b4516a7d9f70580d3d9c6 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:46.978626 22185 raft_consensus.cc:729] T 4b7d0daf418b4516a7d9f70580d3d9c6 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:46.979086 13185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34188b6f8c654108aead589fe01f52d4" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770"
I20240503 13:56:46.979211 13185 raft_consensus.cc:3037] T 34188b6f8c654108aead589fe01f52d4 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:46.978763 22185 consensus_queue.cc:260] T 4b7d0daf418b4516a7d9f70580d3d9c6 P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:46.980264 13185 raft_consensus.cc:2454] T 34188b6f8c654108aead589fe01f52d4 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 1.
I20240503 13:56:46.980551 22185 ts_tablet_manager.cc:1398] T 4b7d0daf418b4516a7d9f70580d3d9c6 P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:46.980770 22185 tablet_bootstrap.cc:492] T a62d43de42a449d0af0c65117b53f041 P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
W20240503 13:56:46.980926 15429 leader_election.cc:341] T 34188b6f8c654108aead589fe01f52d4 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279): Illegal state: must be running to vote when last-logged opid is not known
I20240503 13:56:46.981094 15429 leader_election.cc:302] T 34188b6f8c654108aead589fe01f52d4 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61b887f0d40d497badeca5604c1f0546, 8ccf5a4ac450414c9d3579bd5800b770; no voters: ba0194098aa84449ae17e8a0d4c43703
I20240503 13:56:46.981882 13316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b7d0daf418b4516a7d9f70580d3d9c6" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703" is_pre_election: true
I20240503 13:56:46.982000 13316 raft_consensus.cc:2454] T 4b7d0daf418b4516a7d9f70580d3d9c6 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 0.
I20240503 13:56:46.982707 19618 raft_consensus.cc:2781] T 4b7d0daf418b4516a7d9f70580d3d9c6 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:46.982769 19618 raft_consensus.cc:482] T 4b7d0daf418b4516a7d9f70580d3d9c6 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:46.982806 19618 raft_consensus.cc:3037] T 4b7d0daf418b4516a7d9f70580d3d9c6 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:46.983366 22185 tablet_bootstrap.cc:654] T a62d43de42a449d0af0c65117b53f041 P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:46.983826 19618 raft_consensus.cc:504] T 4b7d0daf418b4516a7d9f70580d3d9c6 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:46.984246 19618 leader_election.cc:288] T 4b7d0daf418b4516a7d9f70580d3d9c6 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 election: Requested vote from peers ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279), 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:46.984408 22185 tablet_bootstrap.cc:492] T a62d43de42a449d0af0c65117b53f041 P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:46.984477 22185 ts_tablet_manager.cc:1367] T a62d43de42a449d0af0c65117b53f041 P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 13:56:46.985121 22185 raft_consensus.cc:348] T a62d43de42a449d0af0c65117b53f041 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:46.985672 22185 raft_consensus.cc:374] T a62d43de42a449d0af0c65117b53f041 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:46.985715 22185 raft_consensus.cc:729] T a62d43de42a449d0af0c65117b53f041 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:46.985843 22185 consensus_queue.cc:260] T a62d43de42a449d0af0c65117b53f041 P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:46.986678 22185 ts_tablet_manager.cc:1398] T a62d43de42a449d0af0c65117b53f041 P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:56:46.986655 13051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b7d0daf418b4516a7d9f70580d3d9c6" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e"
I20240503 13:56:46.986780 13051 raft_consensus.cc:3037] T 4b7d0daf418b4516a7d9f70580d3d9c6 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:46.987838 13051 raft_consensus.cc:2454] T 4b7d0daf418b4516a7d9f70580d3d9c6 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 1.
I20240503 13:56:46.988179 15692 leader_election.cc:302] T 4b7d0daf418b4516a7d9f70580d3d9c6 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 76c386bef61a4bd5a907266a5118ffe2; no voters:
I20240503 13:56:46.988339 19618 raft_consensus.cc:2781] T 4b7d0daf418b4516a7d9f70580d3d9c6 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:46.988428 19618 raft_consensus.cc:686] T 4b7d0daf418b4516a7d9f70580d3d9c6 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 LEADER]: Becoming Leader. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Running, Role: LEADER
I20240503 13:56:46.988584 19618 consensus_queue.cc:237] T 4b7d0daf418b4516a7d9f70580d3d9c6 P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:46.989424 13316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b7d0daf418b4516a7d9f70580d3d9c6" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703"
I20240503 13:56:46.989545 13316 raft_consensus.cc:3037] T 4b7d0daf418b4516a7d9f70580d3d9c6 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:46.990953 13316 raft_consensus.cc:2454] T 4b7d0daf418b4516a7d9f70580d3d9c6 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 1.
I20240503 13:56:46.990954 12910 catalog_manager.cc:5484] T 4b7d0daf418b4516a7d9f70580d3d9c6 P 76c386bef61a4bd5a907266a5118ffe2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71). New cstate: current_term: 1 leader_uuid: "76c386bef61a4bd5a907266a5118ffe2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } }
I20240503 13:56:46.991591 20722 raft_consensus.cc:2781] T 34188b6f8c654108aead589fe01f52d4 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:46.991701 20722 raft_consensus.cc:686] T 34188b6f8c654108aead589fe01f52d4 P 61b887f0d40d497badeca5604c1f0546 [term 1 LEADER]: Becoming Leader. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Running, Role: LEADER
I20240503 13:56:46.991850 20722 consensus_queue.cc:237] T 34188b6f8c654108aead589fe01f52d4 P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:46.994212 12910 catalog_manager.cc:5484] T 34188b6f8c654108aead589fe01f52d4 P 61b887f0d40d497badeca5604c1f0546 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61b887f0d40d497badeca5604c1f0546 (127.6.98.69). New cstate: current_term: 1 leader_uuid: "61b887f0d40d497badeca5604c1f0546" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } }
I20240503 13:56:47.014986 13316 consensus_queue.cc:237] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:47.016387 15888 raft_consensus.cc:1261] T 477c33f203f649019eff2b28524fad90 P e47105a0f46e4511b7112e571688a4c7 [term 2 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 13:56:47.016542 13445 raft_consensus.cc:1261] T 477c33f203f649019eff2b28524fad90 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 2 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 13:56:47.016863 21875 consensus_queue.cc:1035] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:56:47.017094 21234 consensus_queue.cc:1035] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:56:47.019462 21981 raft_consensus.cc:482] T a62d43de42a449d0af0c65117b53f041 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:47.019665 21981 raft_consensus.cc:504] T a62d43de42a449d0af0c65117b53f041 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:47.020632 21981 leader_election.cc:288] T a62d43de42a449d0af0c65117b53f041 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:47.020996 13316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a62d43de42a449d0af0c65117b53f041" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703" is_pre_election: true
I20240503 13:56:47.021167 13316 raft_consensus.cc:2454] T a62d43de42a449d0af0c65117b53f041 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 0.
I20240503 13:56:47.021478 15561 leader_election.cc:302] T a62d43de42a449d0af0c65117b53f041 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:47.021701 21981 raft_consensus.cc:2781] T a62d43de42a449d0af0c65117b53f041 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:47.021950 21981 raft_consensus.cc:482] T a62d43de42a449d0af0c65117b53f041 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:47.021996 21981 raft_consensus.cc:3037] T a62d43de42a449d0af0c65117b53f041 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:47.021883 15888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a62d43de42a449d0af0c65117b53f041" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7" is_pre_election: true
I20240503 13:56:47.022070 15888 raft_consensus.cc:2454] T a62d43de42a449d0af0c65117b53f041 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 0.
I20240503 13:56:47.020282 21234 raft_consensus.cc:2932] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:47.022924 15757 raft_consensus.cc:1261] T 3f5d536cffe742e69f4b0d3b93caf76c P 76c386bef61a4bd5a907266a5118ffe2 [term 2 FOLLOWER]: Refusing update from remote peer 60ba3ffe3b7746469834cd91c84be14e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 13:56:47.023236 21981 raft_consensus.cc:504] T a62d43de42a449d0af0c65117b53f041 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:47.023871 13445 raft_consensus.cc:2932] T 477c33f203f649019eff2b28524fad90 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:47.024315 15888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a62d43de42a449d0af0c65117b53f041" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7"
I20240503 13:56:47.024663 15888 raft_consensus.cc:3037] T a62d43de42a449d0af0c65117b53f041 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:47.025106 20870 consensus_queue.cc:1035] T 3f5d536cffe742e69f4b0d3b93caf76c P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:56:47.025197 15889 raft_consensus.cc:2932] T 477c33f203f649019eff2b28524fad90 P e47105a0f46e4511b7112e571688a4c7 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:47.026064 15888 raft_consensus.cc:2454] T a62d43de42a449d0af0c65117b53f041 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 1.
I20240503 13:56:47.027629 15558 leader_election.cc:302] T a62d43de42a449d0af0c65117b53f041 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:47.027901 21979 raft_consensus.cc:2781] T a62d43de42a449d0af0c65117b53f041 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:47.027937 21981 leader_election.cc:288] T a62d43de42a449d0af0c65117b53f041 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 election: Requested vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:47.028076 21979 raft_consensus.cc:686] T a62d43de42a449d0af0c65117b53f041 P 009f849ee7b54c80a3f5264695047e4d [term 1 LEADER]: Becoming Leader. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Running, Role: LEADER
I20240503 13:56:47.028095 12910 catalog_manager.cc:5484] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 reported cstate change: config changed from index 4 to 6, VOTER 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66) evicted. New cstate: current_term: 2 leader_uuid: "ba0194098aa84449ae17e8a0d4c43703" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:47.028229 21979 consensus_queue.cc:237] T a62d43de42a449d0af0c65117b53f041 P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:47.028589 13316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a62d43de42a449d0af0c65117b53f041" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703"
I20240503 13:56:47.028733 13316 raft_consensus.cc:3037] T a62d43de42a449d0af0c65117b53f041 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:47.029774 13316 raft_consensus.cc:2454] T a62d43de42a449d0af0c65117b53f041 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 1.
I20240503 13:56:47.032217 12903 catalog_manager.cc:5484] T a62d43de42a449d0af0c65117b53f041 P 009f849ee7b54c80a3f5264695047e4d reported cstate change: term changed from 0 to 1, leader changed from <none> to 009f849ee7b54c80a3f5264695047e4d (127.6.98.70). New cstate: current_term: 1 leader_uuid: "009f849ee7b54c80a3f5264695047e4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:47.032997 13164 tablet_service.cc:1514] Processing DeleteTablet for tablet 477c33f203f649019eff2b28524fad90 with delete_type TABLET_DATA_TOMBSTONED (TS 8ccf5a4ac450414c9d3579bd5800b770 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41668
I20240503 13:56:47.035044 22224 tablet_replica.cc:330] T 477c33f203f649019eff2b28524fad90 P 8ccf5a4ac450414c9d3579bd5800b770: stopping tablet replica
I20240503 13:56:47.035215 22224 raft_consensus.cc:2229] T 477c33f203f649019eff2b28524fad90 P 8ccf5a4ac450414c9d3579bd5800b770 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:47.035444 22224 raft_consensus.cc:2258] T 477c33f203f649019eff2b28524fad90 P 8ccf5a4ac450414c9d3579bd5800b770 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:47.036057 22224 ts_tablet_manager.cc:1875] T 477c33f203f649019eff2b28524fad90 P 8ccf5a4ac450414c9d3579bd5800b770: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:56:47.038633 22224 ts_tablet_manager.cc:1888] T 477c33f203f649019eff2b28524fad90 P 8ccf5a4ac450414c9d3579bd5800b770: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 13:56:47.038745 22224 log.cc:1196] T 477c33f203f649019eff2b28524fad90 P 8ccf5a4ac450414c9d3579bd5800b770: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/wal/wals/477c33f203f649019eff2b28524fad90
I20240503 13:56:47.039232 12899 catalog_manager.cc:4830] TS 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387): tablet 477c33f203f649019eff2b28524fad90 (table rebalance_test_table_1 [id=ba90ff8253204f25829d57a56c0b5e94]) successfully deleted
I20240503 13:56:47.066787 22222 raft_consensus.cc:482] T 1307b1fe211b4fd3947d1a7361cfd562 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:47.066919 22222 raft_consensus.cc:504] T 1307b1fe211b4fd3947d1a7361cfd562 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:47.067346 22222 leader_election.cc:288] T 1307b1fe211b4fd3947d1a7361cfd562 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387), 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153)
I20240503 13:56:47.067684 13185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1307b1fe211b4fd3947d1a7361cfd562" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770" is_pre_election: true
I20240503 13:56:47.067746 15493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1307b1fe211b4fd3947d1a7361cfd562" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546" is_pre_election: true
I20240503 13:56:47.067809 13185 raft_consensus.cc:2454] T 1307b1fe211b4fd3947d1a7361cfd562 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 0.
I20240503 13:56:47.067875 15493 raft_consensus.cc:2454] T 1307b1fe211b4fd3947d1a7361cfd562 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 0.
I20240503 13:56:47.068150 15561 leader_election.cc:302] T 1307b1fe211b4fd3947d1a7361cfd562 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 61b887f0d40d497badeca5604c1f0546; no voters:
I20240503 13:56:47.068409 22222 raft_consensus.cc:2781] T 1307b1fe211b4fd3947d1a7361cfd562 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:47.068467 22222 raft_consensus.cc:482] T 1307b1fe211b4fd3947d1a7361cfd562 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:47.068538 22222 raft_consensus.cc:3037] T 1307b1fe211b4fd3947d1a7361cfd562 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:47.069727 22222 raft_consensus.cc:504] T 1307b1fe211b4fd3947d1a7361cfd562 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:47.070015 22222 leader_election.cc:288] T 1307b1fe211b4fd3947d1a7361cfd562 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 election: Requested vote from peers 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387), 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153)
I20240503 13:56:47.070370 13185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1307b1fe211b4fd3947d1a7361cfd562" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770"
I20240503 13:56:47.070494 13185 raft_consensus.cc:3037] T 1307b1fe211b4fd3947d1a7361cfd562 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:47.071455 13185 raft_consensus.cc:2454] T 1307b1fe211b4fd3947d1a7361cfd562 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 1.
I20240503 13:56:47.071693 15493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1307b1fe211b4fd3947d1a7361cfd562" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546"
I20240503 13:56:47.071806 15493 raft_consensus.cc:3037] T 1307b1fe211b4fd3947d1a7361cfd562 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:47.071813 15561 leader_election.cc:302] T 1307b1fe211b4fd3947d1a7361cfd562 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 8ccf5a4ac450414c9d3579bd5800b770; no voters:
I20240503 13:56:47.071950 22222 raft_consensus.cc:2781] T 1307b1fe211b4fd3947d1a7361cfd562 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:47.072023 22222 raft_consensus.cc:686] T 1307b1fe211b4fd3947d1a7361cfd562 P 009f849ee7b54c80a3f5264695047e4d [term 1 LEADER]: Becoming Leader. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Running, Role: LEADER
I20240503 13:56:47.072151 22222 consensus_queue.cc:237] T 1307b1fe211b4fd3947d1a7361cfd562 P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:47.072818 15493 raft_consensus.cc:2454] T 1307b1fe211b4fd3947d1a7361cfd562 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 1.
I20240503 13:56:47.073575 15889 consensus_queue.cc:237] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:47.074805 12911 catalog_manager.cc:5484] T 1307b1fe211b4fd3947d1a7361cfd562 P 009f849ee7b54c80a3f5264695047e4d reported cstate change: term changed from 0 to 1, leader changed from <none> to 009f849ee7b54c80a3f5264695047e4d (127.6.98.70). New cstate: current_term: 1 leader_uuid: "009f849ee7b54c80a3f5264695047e4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:47.075572 15757 raft_consensus.cc:1261] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Refusing update from remote peer e47105a0f46e4511b7112e571688a4c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 13:56:47.075814 15825 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P e47105a0f46e4511b7112e571688a4c7 -> Peer ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279): Couldn't send request to peer ba0194098aa84449ae17e8a0d4c43703. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7bdb9c14caf04abca39e56f8bcf1d1d4. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:47.076436 15623 raft_consensus.cc:1261] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Refusing update from remote peer e47105a0f46e4511b7112e571688a4c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:47.076866 22221 consensus_queue.cc:1035] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:56:47.077196 22221 consensus_queue.cc:1035] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 13:56:47.077421 13363 tablet.cc:2369] T 34188b6f8c654108aead589fe01f52d4 P ba0194098aa84449ae17e8a0d4c43703: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 13:56:47.079520 22253 raft_consensus.cc:2932] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P e47105a0f46e4511b7112e571688a4c7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:47.079927 15757 raft_consensus.cc:2932] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:47.082084 15624 raft_consensus.cc:2932] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:47.083106 12911 catalog_manager.cc:5484] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 76c386bef61a4bd5a907266a5118ffe2 reported cstate change: config changed from index -1 to 2, VOTER ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) added. New cstate: current_term: 1 leader_uuid: "e47105a0f46e4511b7112e571688a4c7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:47.140748 12909 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:56:47.144755 15889 raft_consensus.cc:1261] T a62d43de42a449d0af0c65117b53f041 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Refusing update from remote peer 009f849ee7b54c80a3f5264695047e4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:47.144830 13316 raft_consensus.cc:1261] T a62d43de42a449d0af0c65117b53f041 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Refusing update from remote peer 009f849ee7b54c80a3f5264695047e4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:47.145148 21979 consensus_queue.cc:1035] T a62d43de42a449d0af0c65117b53f041 P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:47.145484 21979 consensus_queue.cc:1035] T a62d43de42a449d0af0c65117b53f041 P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:47.149519 12907 catalog_manager.cc:5576] TS e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753) does not have the latest schema for tablet a62d43de42a449d0af0c65117b53f041 (table rebalancer_dynamic_table [id=4fc2a17ebd4a4e1a8d55eb649fbf6da6]). Expected version 1 got 0
I20240503 13:56:47.150964 22301 tablet.cc:1722] T a62d43de42a449d0af0c65117b53f041 P 009f849ee7b54c80a3f5264695047e4d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:47.151428 12912 catalog_manager.cc:5576] TS ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279) does not have the latest schema for tablet a62d43de42a449d0af0c65117b53f041 (table rebalancer_dynamic_table [id=4fc2a17ebd4a4e1a8d55eb649fbf6da6]). Expected version 1 got 0
I20240503 13:56:47.152066 22301 tablet.cc:1620] T a62d43de42a449d0af0c65117b53f041 P 009f849ee7b54c80a3f5264695047e4d: MemRowSet was empty: no flush needed.
I20240503 13:56:47.156911 12904 catalog_manager.cc:5576] TS ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279) does not have the latest schema for tablet a62d43de42a449d0af0c65117b53f041 (table rebalancer_dynamic_table [id=4fc2a17ebd4a4e1a8d55eb649fbf6da6]). Expected version 1 got 0
I20240503 13:56:47.157302 12911 catalog_manager.cc:5576] TS e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753) does not have the latest schema for tablet a62d43de42a449d0af0c65117b53f041 (table rebalancer_dynamic_table [id=4fc2a17ebd4a4e1a8d55eb649fbf6da6]). Expected version 1 got 0
I20240503 13:56:47.159240 22302 tablet.cc:1722] T a62d43de42a449d0af0c65117b53f041 P e47105a0f46e4511b7112e571688a4c7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:47.161103 22302 tablet.cc:1620] T a62d43de42a449d0af0c65117b53f041 P e47105a0f46e4511b7112e571688a4c7: MemRowSet was empty: no flush needed.
I20240503 13:56:47.164700 15624 raft_consensus.cc:1261] T 3f5d536cffe742e69f4b0d3b93caf76c P 009f849ee7b54c80a3f5264695047e4d [term 2 FOLLOWER]: Refusing update from remote peer 60ba3ffe3b7746469834cd91c84be14e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 13:56:47.165937 20870 consensus_queue.cc:1035] T 3f5d536cffe742e69f4b0d3b93caf76c P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:56:47.170328 12909 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=4fc2a17ebd4a4e1a8d55eb649fbf6da6] alter complete (version 1)
I20240503 13:56:47.170341 13316 raft_consensus.cc:1261] T 3f5d536cffe742e69f4b0d3b93caf76c P ba0194098aa84449ae17e8a0d4c43703 [term 2 FOLLOWER]: Refusing update from remote peer 60ba3ffe3b7746469834cd91c84be14e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 13:56:47.170765 22054 consensus_queue.cc:1035] T 3f5d536cffe742e69f4b0d3b93caf76c P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:56:47.174307 22303 tablet.cc:1722] T a62d43de42a449d0af0c65117b53f041 P ba0194098aa84449ae17e8a0d4c43703: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:47.174741 22303 tablet.cc:1620] T a62d43de42a449d0af0c65117b53f041 P ba0194098aa84449ae17e8a0d4c43703: MemRowSet was empty: no flush needed.
I20240503 13:56:47.209728 15605 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 13:56:47.217849 15473 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 13:56:47.219117 13424 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:47.222695 13032 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 13:56:47.231917 13296 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 13:56:47.243652 13164 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 13:56:47.250420 15738 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 13:56:47.268934 21234 raft_consensus.cc:482] T 34188b6f8c654108aead589fe01f52d4 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:47.269065 21234 raft_consensus.cc:504] T 34188b6f8c654108aead589fe01f52d4 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:47.269443 21234 leader_election.cc:288] T 34188b6f8c654108aead589fe01f52d4 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:47.269845 15493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34188b6f8c654108aead589fe01f52d4" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546" is_pre_election: true
I20240503 13:56:47.270505 13183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34188b6f8c654108aead589fe01f52d4" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770" is_pre_election: true
I20240503 13:56:47.270653 13183 raft_consensus.cc:2379] T 34188b6f8c654108aead589fe01f52d4 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ba0194098aa84449ae17e8a0d4c43703 in current term 1: Already voted for candidate 61b887f0d40d497badeca5604c1f0546 in this term.
I20240503 13:56:47.271157 13252 leader_election.cc:302] T 34188b6f8c654108aead589fe01f52d4 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba0194098aa84449ae17e8a0d4c43703; no voters: 61b887f0d40d497badeca5604c1f0546, 8ccf5a4ac450414c9d3579bd5800b770
I20240503 13:56:47.271374 21234 raft_consensus.cc:3037] T 34188b6f8c654108aead589fe01f52d4 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:47.272425 21234 raft_consensus.cc:2726] T 34188b6f8c654108aead589fe01f52d4 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 13:56:47.275360 12908 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:56:47.284418 22301 tablet.cc:1722] T a62d43de42a449d0af0c65117b53f041 P 009f849ee7b54c80a3f5264695047e4d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:47.284843 22301 tablet.cc:1620] T a62d43de42a449d0af0c65117b53f041 P 009f849ee7b54c80a3f5264695047e4d: MemRowSet was empty: no flush needed.
I20240503 13:56:47.285785 22303 tablet.cc:1722] T a62d43de42a449d0af0c65117b53f041 P ba0194098aa84449ae17e8a0d4c43703: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:47.285890 22302 tablet.cc:1722] T a62d43de42a449d0af0c65117b53f041 P e47105a0f46e4511b7112e571688a4c7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:47.286108 22303 tablet.cc:1620] T a62d43de42a449d0af0c65117b53f041 P ba0194098aa84449ae17e8a0d4c43703: MemRowSet was empty: no flush needed.
I20240503 13:56:47.286208 22302 tablet.cc:1620] T a62d43de42a449d0af0c65117b53f041 P e47105a0f46e4511b7112e571688a4c7: MemRowSet was empty: no flush needed.
I20240503 13:56:47.293675 12908 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=4fc2a17ebd4a4e1a8d55eb649fbf6da6] alter complete (version 2)
I20240503 13:56:47.302363 15867 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:56:47.373356 20722 consensus_queue.cc:1035] T 34188b6f8c654108aead589fe01f52d4 P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:47.376760 21492 consensus_queue.cc:1035] T 34188b6f8c654108aead589fe01f52d4 P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:47.395509 12903 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:56:47.402055 22301 tablet.cc:1722] T a62d43de42a449d0af0c65117b53f041 P 009f849ee7b54c80a3f5264695047e4d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:47.402674 22303 tablet.cc:1722] T a62d43de42a449d0af0c65117b53f041 P ba0194098aa84449ae17e8a0d4c43703: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:47.402915 22302 tablet.cc:1722] T a62d43de42a449d0af0c65117b53f041 P e47105a0f46e4511b7112e571688a4c7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:47.410048 12904 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table_new_name [id=4fc2a17ebd4a4e1a8d55eb649fbf6da6] alter complete (version 3)
I20240503 13:56:47.462177 15605 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 13:56:47.463804 13032 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 13:56:47.469763 13296 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 13:56:47.472637 13164 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 13:56:47.473151 15473 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 13:56:47.481081 15867 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:56:47.491922 13424 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:47.507227 21252 consensus_queue.cc:1035] T 4b7d0daf418b4516a7d9f70580d3d9c6 P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:47.507645 15738 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 13:56:47.512966 12909 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:47.513155 12909 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:47.516027 12909 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Sending DeleteTablet for 3 replicas of tablet a62d43de42a449d0af0c65117b53f041
I20240503 13:56:47.516625 15605 tablet_service.cc:1514] Processing DeleteTablet for tablet a62d43de42a449d0af0c65117b53f041 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:47 UTC) from {username='slave'} at 127.0.0.1:60718
I20240503 13:56:47.516956 13296 tablet_service.cc:1514] Processing DeleteTablet for tablet a62d43de42a449d0af0c65117b53f041 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:47 UTC) from {username='slave'} at 127.0.0.1:33250
I20240503 13:56:47.517237 15867 tablet_service.cc:1514] Processing DeleteTablet for tablet a62d43de42a449d0af0c65117b53f041 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:47 UTC) from {username='slave'} at 127.0.0.1:50498
I20240503 13:56:47.517798 22395 tablet_replica.cc:330] T a62d43de42a449d0af0c65117b53f041 P 009f849ee7b54c80a3f5264695047e4d: stopping tablet replica
I20240503 13:56:47.518000 22397 tablet_replica.cc:330] T a62d43de42a449d0af0c65117b53f041 P e47105a0f46e4511b7112e571688a4c7: stopping tablet replica
I20240503 13:56:47.518174 22397 raft_consensus.cc:2229] T a62d43de42a449d0af0c65117b53f041 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:47.518352 22395 raft_consensus.cc:2229] T a62d43de42a449d0af0c65117b53f041 P 009f849ee7b54c80a3f5264695047e4d [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:56:47.518647 22397 raft_consensus.cc:2258] T a62d43de42a449d0af0c65117b53f041 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:47.518850 22395 raft_consensus.cc:2258] T a62d43de42a449d0af0c65117b53f041 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:47.519762 22397 ts_tablet_manager.cc:1875] T a62d43de42a449d0af0c65117b53f041 P e47105a0f46e4511b7112e571688a4c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:47.519970 22395 ts_tablet_manager.cc:1875] T a62d43de42a449d0af0c65117b53f041 P 009f849ee7b54c80a3f5264695047e4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:47.520673 21979 consensus_queue.cc:1035] T 1307b1fe211b4fd3947d1a7361cfd562 P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:47.522189 22397 ts_tablet_manager.cc:1888] T a62d43de42a449d0af0c65117b53f041 P e47105a0f46e4511b7112e571688a4c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:47.524577 22397 log.cc:1196] T a62d43de42a449d0af0c65117b53f041 P e47105a0f46e4511b7112e571688a4c7: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-7/wal/wals/a62d43de42a449d0af0c65117b53f041
I20240503 13:56:47.524957 22397 ts_tablet_manager.cc:1909] T a62d43de42a449d0af0c65117b53f041 P e47105a0f46e4511b7112e571688a4c7: Deleting consensus metadata
I20240503 13:56:47.525946 12896 catalog_manager.cc:4830] TS e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): tablet a62d43de42a449d0af0c65117b53f041 (table rebalancer_dynamic_table_new_name [id=4fc2a17ebd4a4e1a8d55eb649fbf6da6]) successfully deleted
I20240503 13:56:47.526738 22395 ts_tablet_manager.cc:1888] T a62d43de42a449d0af0c65117b53f041 P 009f849ee7b54c80a3f5264695047e4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:47.526826 22395 log.cc:1196] T a62d43de42a449d0af0c65117b53f041 P 009f849ee7b54c80a3f5264695047e4d: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/wal/wals/a62d43de42a449d0af0c65117b53f041
I20240503 13:56:47.527197 22395 ts_tablet_manager.cc:1909] T a62d43de42a449d0af0c65117b53f041 P 009f849ee7b54c80a3f5264695047e4d: Deleting consensus metadata
I20240503 13:56:47.528129 12896 catalog_manager.cc:4830] TS 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951): tablet a62d43de42a449d0af0c65117b53f041 (table rebalancer_dynamic_table_new_name [id=4fc2a17ebd4a4e1a8d55eb649fbf6da6]) successfully deleted
I20240503 13:56:47.532553 22396 tablet_replica.cc:330] T a62d43de42a449d0af0c65117b53f041 P ba0194098aa84449ae17e8a0d4c43703: stopping tablet replica
I20240503 13:56:47.532716 22396 raft_consensus.cc:2229] T a62d43de42a449d0af0c65117b53f041 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:47.532694 22222 consensus_queue.cc:1035] T 1307b1fe211b4fd3947d1a7361cfd562 P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:47.533448 22396 raft_consensus.cc:2258] T a62d43de42a449d0af0c65117b53f041 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:47.535949 22396 ts_tablet_manager.cc:1875] T a62d43de42a449d0af0c65117b53f041 P ba0194098aa84449ae17e8a0d4c43703: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:47.538013 21252 consensus_queue.cc:1035] T 4b7d0daf418b4516a7d9f70580d3d9c6 P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:47.538569 22396 ts_tablet_manager.cc:1888] T a62d43de42a449d0af0c65117b53f041 P ba0194098aa84449ae17e8a0d4c43703: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:47.542150 22396 log.cc:1196] T a62d43de42a449d0af0c65117b53f041 P ba0194098aa84449ae17e8a0d4c43703: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/wal/wals/a62d43de42a449d0af0c65117b53f041
I20240503 13:56:47.543264 22396 ts_tablet_manager.cc:1909] T a62d43de42a449d0af0c65117b53f041 P ba0194098aa84449ae17e8a0d4c43703: Deleting consensus metadata
I20240503 13:56:47.543947 12899 catalog_manager.cc:4830] TS ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279): tablet a62d43de42a449d0af0c65117b53f041 (table rebalancer_dynamic_table_new_name [id=4fc2a17ebd4a4e1a8d55eb649fbf6da6]) successfully deleted
I20240503 13:56:47.568075 13183 consensus_queue.cc:237] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:47.569437 15888 raft_consensus.cc:1261] T cbd6d38ea18b484d80f4a918148efd3a P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Refusing update from remote peer 8ccf5a4ac450414c9d3579bd5800b770: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:56:47.569545 15493 raft_consensus.cc:1261] T cbd6d38ea18b484d80f4a918148efd3a P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Refusing update from remote peer 8ccf5a4ac450414c9d3579bd5800b770: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:56:47.569824 22418 consensus_queue.cc:1035] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:56:47.570044 22190 consensus_queue.cc:1035] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:56:47.571833 22190 raft_consensus.cc:2932] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:47.572237 15888 raft_consensus.cc:2932] T cbd6d38ea18b484d80f4a918148efd3a P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:47.573143 15493 raft_consensus.cc:2932] T cbd6d38ea18b484d80f4a918148efd3a P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:47.575206 12906 catalog_manager.cc:5484] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 reported cstate change: config changed from index 2 to 3, VOTER 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71) evicted. New cstate: current_term: 1 leader_uuid: "8ccf5a4ac450414c9d3579bd5800b770" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:47.578552 15738 tablet_service.cc:1514] Processing DeleteTablet for tablet cbd6d38ea18b484d80f4a918148efd3a with delete_type TABLET_DATA_TOMBSTONED (TS 76c386bef61a4bd5a907266a5118ffe2 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:44984
I20240503 13:56:47.578917 22423 tablet_replica.cc:330] T cbd6d38ea18b484d80f4a918148efd3a P 76c386bef61a4bd5a907266a5118ffe2: stopping tablet replica
I20240503 13:56:47.579138 22423 raft_consensus.cc:2229] T cbd6d38ea18b484d80f4a918148efd3a P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:47.579540 22423 raft_consensus.cc:2258] T cbd6d38ea18b484d80f4a918148efd3a P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:47.580240 22423 ts_tablet_manager.cc:1875] T cbd6d38ea18b484d80f4a918148efd3a P 76c386bef61a4bd5a907266a5118ffe2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:56:47.583034 22423 ts_tablet_manager.cc:1888] T cbd6d38ea18b484d80f4a918148efd3a P 76c386bef61a4bd5a907266a5118ffe2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 13:56:47.583101 22423 log.cc:1196] T cbd6d38ea18b484d80f4a918148efd3a P 76c386bef61a4bd5a907266a5118ffe2: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/wal/wals/cbd6d38ea18b484d80f4a918148efd3a
I20240503 13:56:47.583544 12899 catalog_manager.cc:4830] TS 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885): tablet cbd6d38ea18b484d80f4a918148efd3a (table rebalancer_extra_table_12 [id=46a3f42046eb4cb89a40f991a54af62b]) successfully deleted
I20240503 13:56:47.602257 12909 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_extra_table_48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:56:47.608795 13424 tablet_service.cc:1467] Processing CreateTablet for tablet e749c1924f504c6f8f780b71865ab0ab (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=314c8e36cd5d4a1aae8b2d5220411838]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:47.608795 13164 tablet_service.cc:1467] Processing CreateTablet for tablet e749c1924f504c6f8f780b71865ab0ab (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=314c8e36cd5d4a1aae8b2d5220411838]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:47.609104 13424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e749c1924f504c6f8f780b71865ab0ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:47.609104 13164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e749c1924f504c6f8f780b71865ab0ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:47.611371 22440 tablet_bootstrap.cc:492] T e749c1924f504c6f8f780b71865ab0ab P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:47.611392 22441 tablet_bootstrap.cc:492] T e749c1924f504c6f8f780b71865ab0ab P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:47.611785 15473 tablet_service.cc:1467] Processing CreateTablet for tablet 289d31dd1f8f40d9b87d7ad401d2d8dd (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=314c8e36cd5d4a1aae8b2d5220411838]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:47.612053 15473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 289d31dd1f8f40d9b87d7ad401d2d8dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:47.612013 15867 tablet_service.cc:1467] Processing CreateTablet for tablet 289d31dd1f8f40d9b87d7ad401d2d8dd (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=314c8e36cd5d4a1aae8b2d5220411838]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:47.612278 15867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 289d31dd1f8f40d9b87d7ad401d2d8dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:47.612388 22441 tablet_bootstrap.cc:654] T e749c1924f504c6f8f780b71865ab0ab P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:47.613585 15736 tablet_service.cc:1467] Processing CreateTablet for tablet 867500b772894823b3c240a230c4dfda (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=314c8e36cd5d4a1aae8b2d5220411838]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:47.613839 15736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 867500b772894823b3c240a230c4dfda. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:47.614082 22443 tablet_bootstrap.cc:492] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:47.614313 15472 tablet_service.cc:1467] Processing CreateTablet for tablet 867500b772894823b3c240a230c4dfda (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=314c8e36cd5d4a1aae8b2d5220411838]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:47.614535 15472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 867500b772894823b3c240a230c4dfda. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:47.614830 13295 tablet_service.cc:1467] Processing CreateTablet for tablet 867500b772894823b3c240a230c4dfda (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=314c8e36cd5d4a1aae8b2d5220411838]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:47.614970 22443 tablet_bootstrap.cc:654] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:47.615116 13295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 867500b772894823b3c240a230c4dfda. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:47.615976 22443 tablet_bootstrap.cc:492] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546: No bootstrap required, opened a new log
I20240503 13:56:47.616062 22443 ts_tablet_manager.cc:1367] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:47.617450 13296 tablet_service.cc:1467] Processing CreateTablet for tablet 289d31dd1f8f40d9b87d7ad401d2d8dd (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=314c8e36cd5d4a1aae8b2d5220411838]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:47.617411 15738 tablet_service.cc:1467] Processing CreateTablet for tablet e749c1924f504c6f8f780b71865ab0ab (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=314c8e36cd5d4a1aae8b2d5220411838]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:47.618009 15738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e749c1924f504c6f8f780b71865ab0ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:47.618474 22447 tablet_bootstrap.cc:492] T 867500b772894823b3c240a230c4dfda P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:47.618373 12909 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:56:47.619146 22446 tablet_bootstrap.cc:492] T 867500b772894823b3c240a230c4dfda P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:47.620414 22447 tablet_bootstrap.cc:654] T 867500b772894823b3c240a230c4dfda P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:47.620841 13296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 289d31dd1f8f40d9b87d7ad401d2d8dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:47.621404 22446 tablet_bootstrap.cc:654] T 867500b772894823b3c240a230c4dfda P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:47.625418 22446 tablet_bootstrap.cc:492] T 867500b772894823b3c240a230c4dfda P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:47.625526 22440 tablet_bootstrap.cc:654] T e749c1924f504c6f8f780b71865ab0ab P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:47.625576 22446 ts_tablet_manager.cc:1367] T 867500b772894823b3c240a230c4dfda P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 13:56:47.625916 22447 tablet_bootstrap.cc:492] T 867500b772894823b3c240a230c4dfda P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:47.626055 22447 ts_tablet_manager.cc:1367] T 867500b772894823b3c240a230c4dfda P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 13:56:47.626248 22443 raft_consensus.cc:348] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:47.626385 22443 raft_consensus.cc:374] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:47.626421 22443 raft_consensus.cc:729] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:47.626538 22443 consensus_queue.cc:260] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:47.626840 22443 ts_tablet_manager.cc:1398] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.011s user 0.000s sys 0.000s
I20240503 13:56:47.627053 22443 tablet_bootstrap.cc:492] T 867500b772894823b3c240a230c4dfda P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:47.627574 22444 tablet_bootstrap.cc:492] T 289d31dd1f8f40d9b87d7ad401d2d8dd P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:47.627929 22443 tablet_bootstrap.cc:654] T 867500b772894823b3c240a230c4dfda P 61b887f0d40d497badeca5604c1f0546: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:47.628314 22446 raft_consensus.cc:348] T 867500b772894823b3c240a230c4dfda P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:47.629838 22447 raft_consensus.cc:348] T 867500b772894823b3c240a230c4dfda P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:47.630013 22447 raft_consensus.cc:374] T 867500b772894823b3c240a230c4dfda P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:47.630069 22447 raft_consensus.cc:729] T 867500b772894823b3c240a230c4dfda P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:47.630234 22447 consensus_queue.cc:260] T 867500b772894823b3c240a230c4dfda P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:47.631053 22444 tablet_bootstrap.cc:654] T 289d31dd1f8f40d9b87d7ad401d2d8dd P e47105a0f46e4511b7112e571688a4c7: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:47.631182 13032 tablet_service.cc:1467] Processing CreateTablet for tablet 4b30d9c772e2480787ec36513f84f3c5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0f7ff4c9597044d68254fb58f84de2bb]), partition=
I20240503 13:56:47.629547 22446 raft_consensus.cc:374] T 867500b772894823b3c240a230c4dfda P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:47.631260 22446 raft_consensus.cc:729] T 867500b772894823b3c240a230c4dfda P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:47.631450 13032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b30d9c772e2480787ec36513f84f3c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:47.631438 22446 consensus_queue.cc:260] T 867500b772894823b3c240a230c4dfda P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:47.631824 22446 ts_tablet_manager.cc:1398] T 867500b772894823b3c240a230c4dfda P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240503 13:56:47.632057 22446 tablet_bootstrap.cc:492] T e749c1924f504c6f8f780b71865ab0ab P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:47.633196 22446 tablet_bootstrap.cc:654] T e749c1924f504c6f8f780b71865ab0ab P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:47.635067 22441 tablet_bootstrap.cc:492] T e749c1924f504c6f8f780b71865ab0ab P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:47.635107 15472 tablet_service.cc:1467] Processing CreateTablet for tablet 4b30d9c772e2480787ec36513f84f3c5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0f7ff4c9597044d68254fb58f84de2bb]), partition=
I20240503 13:56:47.635170 22441 ts_tablet_manager.cc:1367] T e749c1924f504c6f8f780b71865ab0ab P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.000s
I20240503 13:56:47.635269 13424 tablet_service.cc:1467] Processing CreateTablet for tablet 4b30d9c772e2480787ec36513f84f3c5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0f7ff4c9597044d68254fb58f84de2bb]), partition=
I20240503 13:56:47.635334 15472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b30d9c772e2480787ec36513f84f3c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:47.635551 13424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b30d9c772e2480787ec36513f84f3c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:47.635636 22441 raft_consensus.cc:348] T e749c1924f504c6f8f780b71865ab0ab P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:47.635733 22441 raft_consensus.cc:374] T e749c1924f504c6f8f780b71865ab0ab P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:47.635757 22441 raft_consensus.cc:729] T e749c1924f504c6f8f780b71865ab0ab P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:47.635871 22441 consensus_queue.cc:260] T e749c1924f504c6f8f780b71865ab0ab P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:47.636133 22441 ts_tablet_manager.cc:1398] T e749c1924f504c6f8f780b71865ab0ab P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:47.637599 22447 ts_tablet_manager.cc:1398] T 867500b772894823b3c240a230c4dfda P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
I20240503 13:56:47.637782 22447 tablet_bootstrap.cc:492] T 289d31dd1f8f40d9b87d7ad401d2d8dd P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:47.638707 22447 tablet_bootstrap.cc:654] T 289d31dd1f8f40d9b87d7ad401d2d8dd P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:47.639405 22457 tablet_bootstrap.cc:492] T 4b30d9c772e2480787ec36513f84f3c5 P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:47.639771 22447 tablet_bootstrap.cc:492] T 289d31dd1f8f40d9b87d7ad401d2d8dd P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:47.639875 22447 ts_tablet_manager.cc:1367] T 289d31dd1f8f40d9b87d7ad401d2d8dd P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:56:47.640455 22447 raft_consensus.cc:348] T 289d31dd1f8f40d9b87d7ad401d2d8dd P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:47.640619 22447 raft_consensus.cc:374] T 289d31dd1f8f40d9b87d7ad401d2d8dd P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:47.640661 22447 raft_consensus.cc:729] T 289d31dd1f8f40d9b87d7ad401d2d8dd P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:47.640774 22447 consensus_queue.cc:260] T 289d31dd1f8f40d9b87d7ad401d2d8dd P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:47.641002 22447 ts_tablet_manager.cc:1398] T 289d31dd1f8f40d9b87d7ad401d2d8dd P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:47.642850 22457 tablet_bootstrap.cc:654] T 4b30d9c772e2480787ec36513f84f3c5 P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:47.644518 22446 tablet_bootstrap.cc:492] T e749c1924f504c6f8f780b71865ab0ab P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:47.644718 22443 tablet_bootstrap.cc:492] T 867500b772894823b3c240a230c4dfda P 61b887f0d40d497badeca5604c1f0546: No bootstrap required, opened a new log
I20240503 13:56:47.644769 22457 tablet_bootstrap.cc:492] T 4b30d9c772e2480787ec36513f84f3c5 P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:47.644809 22443 ts_tablet_manager.cc:1367] T 867500b772894823b3c240a230c4dfda P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.001s
I20240503 13:56:47.644851 22457 ts_tablet_manager.cc:1367] T 4b30d9c772e2480787ec36513f84f3c5 P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 13:56:47.644701 22446 ts_tablet_manager.cc:1367] T e749c1924f504c6f8f780b71865ab0ab P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.002s
I20240503 13:56:47.645272 22443 raft_consensus.cc:348] T 867500b772894823b3c240a230c4dfda P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:47.645390 22443 raft_consensus.cc:374] T 867500b772894823b3c240a230c4dfda P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:47.645421 22443 raft_consensus.cc:729] T 867500b772894823b3c240a230c4dfda P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:47.645545 22443 consensus_queue.cc:260] T 867500b772894823b3c240a230c4dfda P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:47.646260 22443 ts_tablet_manager.cc:1398] T 867500b772894823b3c240a230c4dfda P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:47.646296 22446 raft_consensus.cc:348] T e749c1924f504c6f8f780b71865ab0ab P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:47.646468 22446 raft_consensus.cc:374] T e749c1924f504c6f8f780b71865ab0ab P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:47.646522 22446 raft_consensus.cc:729] T e749c1924f504c6f8f780b71865ab0ab P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:47.646677 22446 consensus_queue.cc:260] T e749c1924f504c6f8f780b71865ab0ab P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:47.647024 22446 ts_tablet_manager.cc:1398] T e749c1924f504c6f8f780b71865ab0ab P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:56:47.648250 22443 tablet_bootstrap.cc:492] T 4b30d9c772e2480787ec36513f84f3c5 P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:47.648350 22440 tablet_bootstrap.cc:492] T e749c1924f504c6f8f780b71865ab0ab P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:47.648448 22440 ts_tablet_manager.cc:1367] T e749c1924f504c6f8f780b71865ab0ab P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.037s user 0.002s sys 0.000s
I20240503 13:56:47.648957 22440 raft_consensus.cc:348] T e749c1924f504c6f8f780b71865ab0ab P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:47.649058 22440 raft_consensus.cc:374] T e749c1924f504c6f8f780b71865ab0ab P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:47.649088 22440 raft_consensus.cc:729] T e749c1924f504c6f8f780b71865ab0ab P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:47.649200 22440 consensus_queue.cc:260] T e749c1924f504c6f8f780b71865ab0ab P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:47.649459 22440 ts_tablet_manager.cc:1398] T e749c1924f504c6f8f780b71865ab0ab P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:47.649602 22440 tablet_bootstrap.cc:492] T 4b30d9c772e2480787ec36513f84f3c5 P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:47.649911 22443 tablet_bootstrap.cc:654] T 4b30d9c772e2480787ec36513f84f3c5 P 61b887f0d40d497badeca5604c1f0546: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:47.650393 22440 tablet_bootstrap.cc:654] T 4b30d9c772e2480787ec36513f84f3c5 P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:47.651494 22444 tablet_bootstrap.cc:492] T 289d31dd1f8f40d9b87d7ad401d2d8dd P e47105a0f46e4511b7112e571688a4c7: No bootstrap required, opened a new log
I20240503 13:56:47.651569 22443 tablet_bootstrap.cc:492] T 4b30d9c772e2480787ec36513f84f3c5 P 61b887f0d40d497badeca5604c1f0546: No bootstrap required, opened a new log
I20240503 13:56:47.651595 22444 ts_tablet_manager.cc:1367] T 289d31dd1f8f40d9b87d7ad401d2d8dd P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.000s
I20240503 13:56:47.651640 22443 ts_tablet_manager.cc:1367] T 4b30d9c772e2480787ec36513f84f3c5 P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 13:56:47.652016 22444 raft_consensus.cc:348] T 289d31dd1f8f40d9b87d7ad401d2d8dd P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:47.652117 22444 raft_consensus.cc:374] T 289d31dd1f8f40d9b87d7ad401d2d8dd P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:47.652067 22443 raft_consensus.cc:348] T 4b30d9c772e2480787ec36513f84f3c5 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:47.652148 22444 raft_consensus.cc:729] T 289d31dd1f8f40d9b87d7ad401d2d8dd P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:47.652172 22443 raft_consensus.cc:374] T 4b30d9c772e2480787ec36513f84f3c5 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:47.652204 22443 raft_consensus.cc:729] T 4b30d9c772e2480787ec36513f84f3c5 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:47.652264 22444 consensus_queue.cc:260] T 289d31dd1f8f40d9b87d7ad401d2d8dd P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:47.652307 22443 consensus_queue.cc:260] T 4b30d9c772e2480787ec36513f84f3c5 P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:47.652575 22444 ts_tablet_manager.cc:1398] T 289d31dd1f8f40d9b87d7ad401d2d8dd P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:47.652665 22443 ts_tablet_manager.cc:1398] T 4b30d9c772e2480787ec36513f84f3c5 P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:47.650571 22457 raft_consensus.cc:348] T 4b30d9c772e2480787ec36513f84f3c5 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:47.652940 22457 raft_consensus.cc:374] T 4b30d9c772e2480787ec36513f84f3c5 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:47.653020 22457 raft_consensus.cc:729] T 4b30d9c772e2480787ec36513f84f3c5 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:47.653285 22457 consensus_queue.cc:260] T 4b30d9c772e2480787ec36513f84f3c5 P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:47.656628 22440 tablet_bootstrap.cc:492] T 4b30d9c772e2480787ec36513f84f3c5 P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:47.656726 22440 ts_tablet_manager.cc:1367] T 4b30d9c772e2480787ec36513f84f3c5 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 13:56:47.657181 22440 raft_consensus.cc:348] T 4b30d9c772e2480787ec36513f84f3c5 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:47.657743 22440 raft_consensus.cc:374] T 4b30d9c772e2480787ec36513f84f3c5 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:47.657833 22440 raft_consensus.cc:729] T 4b30d9c772e2480787ec36513f84f3c5 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:47.658041 22440 consensus_queue.cc:260] T 4b30d9c772e2480787ec36513f84f3c5 P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:47.658205 22457 ts_tablet_manager.cc:1398] T 4b30d9c772e2480787ec36513f84f3c5 P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.013s user 0.000s sys 0.002s
I20240503 13:56:47.658398 22440 ts_tablet_manager.cc:1398] T 4b30d9c772e2480787ec36513f84f3c5 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:47.680187 22484 ts_tablet_manager.cc:902] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P ba0194098aa84449ae17e8a0d4c43703: Initiating tablet copy from peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:47.681023 22484 tablet_copy_client.cc:308] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P ba0194098aa84449ae17e8a0d4c43703: tablet copy: Beginning tablet copy session from remote peer at address 127.6.98.72:44753
I20240503 13:56:47.682108 15909 tablet_copy_service.cc:140] P e47105a0f46e4511b7112e571688a4c7: Received BeginTabletCopySession request for tablet 7bdb9c14caf04abca39e56f8bcf1d1d4 from peer ba0194098aa84449ae17e8a0d4c43703 ({username='slave'} at 127.6.98.67:56843)
I20240503 13:56:47.682228 15909 tablet_copy_service.cc:161] P e47105a0f46e4511b7112e571688a4c7: Beginning new tablet copy session on tablet 7bdb9c14caf04abca39e56f8bcf1d1d4 from peer ba0194098aa84449ae17e8a0d4c43703 at {username='slave'} at 127.6.98.67:56843: session id = ba0194098aa84449ae17e8a0d4c43703-7bdb9c14caf04abca39e56f8bcf1d1d4
I20240503 13:56:47.683128 15909 tablet_copy_source_session.cc:205] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P e47105a0f46e4511b7112e571688a4c7: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:56:47.685206 22484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bdb9c14caf04abca39e56f8bcf1d1d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:47.687712 22484 tablet_copy_client.cc:783] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P ba0194098aa84449ae17e8a0d4c43703: tablet copy: Starting download of 0 data blocks...
I20240503 13:56:47.687880 22484 tablet_copy_client.cc:647] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P ba0194098aa84449ae17e8a0d4c43703: tablet copy: Starting download of 1 WAL segments...
I20240503 13:56:47.689226 22484 tablet_copy_client.cc:523] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P ba0194098aa84449ae17e8a0d4c43703: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:56:47.690497 22484 tablet_bootstrap.cc:492] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:47.693701 20120 raft_consensus.cc:482] T 4b30d9c772e2480787ec36513f84f3c5 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:47.693805 20120 raft_consensus.cc:504] T 4b30d9c772e2480787ec36513f84f3c5 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:47.694185 20120 leader_election.cc:288] T 4b30d9c772e2480787ec36513f84f3c5 P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153), 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:47.694527 15493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b30d9c772e2480787ec36513f84f3c5" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546" is_pre_election: true
I20240503 13:56:47.694561 13051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b30d9c772e2480787ec36513f84f3c5" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e" is_pre_election: true
I20240503 13:56:47.694684 13051 raft_consensus.cc:2454] T 4b30d9c772e2480787ec36513f84f3c5 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 0.
I20240503 13:56:47.694682 15493 raft_consensus.cc:2454] T 4b30d9c772e2480787ec36513f84f3c5 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 0.
I20240503 13:56:47.694962 13381 leader_election.cc:302] T 4b30d9c772e2480787ec36513f84f3c5 P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 7690ee79577c45e3a2c8f8b552c0bff8; no voters:
I20240503 13:56:47.695169 20120 raft_consensus.cc:2781] T 4b30d9c772e2480787ec36513f84f3c5 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:47.695225 20120 raft_consensus.cc:482] T 4b30d9c772e2480787ec36513f84f3c5 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:47.695266 20120 raft_consensus.cc:3037] T 4b30d9c772e2480787ec36513f84f3c5 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:47.696400 20120 raft_consensus.cc:504] T 4b30d9c772e2480787ec36513f84f3c5 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:47.696781 20120 leader_election.cc:288] T 4b30d9c772e2480787ec36513f84f3c5 P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 election: Requested vote from peers 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153), 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:47.698407 13051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b30d9c772e2480787ec36513f84f3c5" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e"
I20240503 13:56:47.698522 13051 raft_consensus.cc:3037] T 4b30d9c772e2480787ec36513f84f3c5 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:47.699590 13051 raft_consensus.cc:2454] T 4b30d9c772e2480787ec36513f84f3c5 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 1.
I20240503 13:56:47.699965 13381 leader_election.cc:302] T 4b30d9c772e2480787ec36513f84f3c5 P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 7690ee79577c45e3a2c8f8b552c0bff8; no voters:
I20240503 13:56:47.700116 20120 raft_consensus.cc:2781] T 4b30d9c772e2480787ec36513f84f3c5 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:47.700284 20120 raft_consensus.cc:686] T 4b30d9c772e2480787ec36513f84f3c5 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 LEADER]: Becoming Leader. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Running, Role: LEADER
I20240503 13:56:47.700425 20120 consensus_queue.cc:237] T 4b30d9c772e2480787ec36513f84f3c5 P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:47.700973 15493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b30d9c772e2480787ec36513f84f3c5" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546"
I20240503 13:56:47.701107 15493 raft_consensus.cc:3037] T 4b30d9c772e2480787ec36513f84f3c5 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:47.702143 15493 raft_consensus.cc:2454] T 4b30d9c772e2480787ec36513f84f3c5 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 1.
I20240503 13:56:47.705559 12906 catalog_manager.cc:5484] T 4b30d9c772e2480787ec36513f84f3c5 P 7690ee79577c45e3a2c8f8b552c0bff8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68). New cstate: current_term: 1 leader_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } }
I20240503 13:56:47.708338 22422 raft_consensus.cc:482] T e749c1924f504c6f8f780b71865ab0ab P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:47.708443 22422 raft_consensus.cc:504] T e749c1924f504c6f8f780b71865ab0ab P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:47.708914 22422 leader_election.cc:288] T e749c1924f504c6f8f780b71865ab0ab P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885)
I20240503 13:56:47.709162 22484 tablet_bootstrap.cc:492] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P ba0194098aa84449ae17e8a0d4c43703: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:56:47.709396 22484 tablet_bootstrap.cc:492] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P ba0194098aa84449ae17e8a0d4c43703: Bootstrap complete.
I20240503 13:56:47.709415 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e749c1924f504c6f8f780b71865ab0ab" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
I20240503 13:56:47.709551 22484 ts_tablet_manager.cc:1367] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.003s
I20240503 13:56:47.709570 13447 raft_consensus.cc:2454] T e749c1924f504c6f8f780b71865ab0ab P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 0.
I20240503 13:56:47.709954 22484 raft_consensus.cc:348] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:47.710067 22484 raft_consensus.cc:729] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:47.710212 22484 consensus_queue.cc:260] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:47.710500 22484 ts_tablet_manager.cc:1398] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240503 13:56:47.711097 15909 tablet_copy_service.cc:342] P e47105a0f46e4511b7112e571688a4c7: Request end of tablet copy session ba0194098aa84449ae17e8a0d4c43703-7bdb9c14caf04abca39e56f8bcf1d1d4 received from {username='slave'} at 127.6.98.67:56843
I20240503 13:56:47.711179 15909 tablet_copy_service.cc:433] P e47105a0f46e4511b7112e571688a4c7: ending tablet copy session ba0194098aa84449ae17e8a0d4c43703-7bdb9c14caf04abca39e56f8bcf1d1d4 on tablet 7bdb9c14caf04abca39e56f8bcf1d1d4 with peer ba0194098aa84449ae17e8a0d4c43703
I20240503 13:56:47.711216 13118 leader_election.cc:302] T e749c1924f504c6f8f780b71865ab0ab P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, 8ccf5a4ac450414c9d3579bd5800b770; no voters:
I20240503 13:56:47.711246 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e749c1924f504c6f8f780b71865ab0ab" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2" is_pre_election: true
I20240503 13:56:47.711359 15757 raft_consensus.cc:2454] T e749c1924f504c6f8f780b71865ab0ab P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 0.
I20240503 13:56:47.713457 22422 raft_consensus.cc:2781] T e749c1924f504c6f8f780b71865ab0ab P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:47.713526 22422 raft_consensus.cc:482] T e749c1924f504c6f8f780b71865ab0ab P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:47.713574 22422 raft_consensus.cc:3037] T e749c1924f504c6f8f780b71865ab0ab P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:47.714725 22422 raft_consensus.cc:504] T e749c1924f504c6f8f780b71865ab0ab P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:47.715121 22422 leader_election.cc:288] T e749c1924f504c6f8f780b71865ab0ab P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 election: Requested vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885)
I20240503 13:56:47.715870 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e749c1924f504c6f8f780b71865ab0ab" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
I20240503 13:56:47.716001 13447 raft_consensus.cc:3037] T e749c1924f504c6f8f780b71865ab0ab P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:47.717038 13447 raft_consensus.cc:2454] T e749c1924f504c6f8f780b71865ab0ab P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 1.
I20240503 13:56:47.717885 13118 leader_election.cc:302] T e749c1924f504c6f8f780b71865ab0ab P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, 8ccf5a4ac450414c9d3579bd5800b770; no voters:
I20240503 13:56:47.718564 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e749c1924f504c6f8f780b71865ab0ab" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2"
I20240503 13:56:47.718678 15757 raft_consensus.cc:3037] T e749c1924f504c6f8f780b71865ab0ab P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:47.718922 22422 raft_consensus.cc:2781] T e749c1924f504c6f8f780b71865ab0ab P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:47.719014 22422 raft_consensus.cc:686] T e749c1924f504c6f8f780b71865ab0ab P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 LEADER]: Becoming Leader. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Running, Role: LEADER
I20240503 13:56:47.719153 22422 consensus_queue.cc:237] T e749c1924f504c6f8f780b71865ab0ab P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:47.719694 15757 raft_consensus.cc:2454] T e749c1924f504c6f8f780b71865ab0ab P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 1.
I20240503 13:56:47.721482 12910 catalog_manager.cc:5484] T e749c1924f504c6f8f780b71865ab0ab P 8ccf5a4ac450414c9d3579bd5800b770 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66). New cstate: current_term: 1 leader_uuid: "8ccf5a4ac450414c9d3579bd5800b770" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:47.773120 22400 raft_consensus.cc:482] T 867500b772894823b3c240a230c4dfda P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:47.773258 22400 raft_consensus.cc:504] T 867500b772894823b3c240a230c4dfda P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:47.773738 22400 leader_election.cc:288] T 867500b772894823b3c240a230c4dfda P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279), 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153)
I20240503 13:56:47.774185 15493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867500b772894823b3c240a230c4dfda" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546" is_pre_election: true
I20240503 13:56:47.774204 13316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867500b772894823b3c240a230c4dfda" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703" is_pre_election: true
I20240503 13:56:47.774317 15493 raft_consensus.cc:2454] T 867500b772894823b3c240a230c4dfda P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 0.
I20240503 13:56:47.774374 13316 raft_consensus.cc:2454] T 867500b772894823b3c240a230c4dfda P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 0.
I20240503 13:56:47.774611 15694 leader_election.cc:302] T 867500b772894823b3c240a230c4dfda P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61b887f0d40d497badeca5604c1f0546, 76c386bef61a4bd5a907266a5118ffe2; no voters:
I20240503 13:56:47.774777 22400 raft_consensus.cc:2781] T 867500b772894823b3c240a230c4dfda P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:47.774837 22400 raft_consensus.cc:482] T 867500b772894823b3c240a230c4dfda P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:47.774863 22400 raft_consensus.cc:3037] T 867500b772894823b3c240a230c4dfda P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:47.775869 22400 raft_consensus.cc:504] T 867500b772894823b3c240a230c4dfda P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:47.776218 22400 leader_election.cc:288] T 867500b772894823b3c240a230c4dfda P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 election: Requested vote from peers ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279), 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153)
I20240503 13:56:47.776609 13316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867500b772894823b3c240a230c4dfda" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703"
I20240503 13:56:47.776615 15493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867500b772894823b3c240a230c4dfda" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546"
I20240503 13:56:47.776731 15493 raft_consensus.cc:3037] T 867500b772894823b3c240a230c4dfda P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:47.776732 13316 raft_consensus.cc:3037] T 867500b772894823b3c240a230c4dfda P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:47.777551 15493 raft_consensus.cc:2454] T 867500b772894823b3c240a230c4dfda P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 1.
I20240503 13:56:47.777638 13316 raft_consensus.cc:2454] T 867500b772894823b3c240a230c4dfda P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 1.
I20240503 13:56:47.777812 15694 leader_election.cc:302] T 867500b772894823b3c240a230c4dfda P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61b887f0d40d497badeca5604c1f0546, 76c386bef61a4bd5a907266a5118ffe2; no voters:
I20240503 13:56:47.777969 22400 raft_consensus.cc:2781] T 867500b772894823b3c240a230c4dfda P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:47.778055 22400 raft_consensus.cc:686] T 867500b772894823b3c240a230c4dfda P 76c386bef61a4bd5a907266a5118ffe2 [term 1 LEADER]: Becoming Leader. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Running, Role: LEADER
I20240503 13:56:47.778203 22400 consensus_queue.cc:237] T 867500b772894823b3c240a230c4dfda P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:47.780797 12906 catalog_manager.cc:5484] T 867500b772894823b3c240a230c4dfda P 76c386bef61a4bd5a907266a5118ffe2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71). New cstate: current_term: 1 leader_uuid: "76c386bef61a4bd5a907266a5118ffe2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:47.823612 12906 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:56:47.828212 13051 raft_consensus.cc:1261] T 4b30d9c772e2480787ec36513f84f3c5 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Refusing update from remote peer 7690ee79577c45e3a2c8f8b552c0bff8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:47.828372 15493 raft_consensus.cc:1261] T 4b30d9c772e2480787ec36513f84f3c5 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Refusing update from remote peer 7690ee79577c45e3a2c8f8b552c0bff8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:47.832953 20120 consensus_queue.cc:1035] T 4b30d9c772e2480787ec36513f84f3c5 P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:47.833326 20120 consensus_queue.cc:1035] T 4b30d9c772e2480787ec36513f84f3c5 P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:47.836789 12911 catalog_manager.cc:5576] TS 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153) does not have the latest schema for tablet 4b30d9c772e2480787ec36513f84f3c5 (table rebalancer_dynamic_table [id=0f7ff4c9597044d68254fb58f84de2bb]). Expected version 1 got 0
I20240503 13:56:47.838263 12911 catalog_manager.cc:5576] TS 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695) does not have the latest schema for tablet 4b30d9c772e2480787ec36513f84f3c5 (table rebalancer_dynamic_table [id=0f7ff4c9597044d68254fb58f84de2bb]). Expected version 1 got 0
I20240503 13:56:47.838877 22498 tablet.cc:1722] T 4b30d9c772e2480787ec36513f84f3c5 P 7690ee79577c45e3a2c8f8b552c0bff8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:47.839255 22498 tablet.cc:1620] T 4b30d9c772e2480787ec36513f84f3c5 P 7690ee79577c45e3a2c8f8b552c0bff8: MemRowSet was empty: no flush needed.
I20240503 13:56:47.839918 22499 tablet.cc:1722] T 4b30d9c772e2480787ec36513f84f3c5 P 61b887f0d40d497badeca5604c1f0546: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:47.840250 22499 tablet.cc:1620] T 4b30d9c772e2480787ec36513f84f3c5 P 61b887f0d40d497badeca5604c1f0546: MemRowSet was empty: no flush needed.
I20240503 13:56:47.840888 22500 tablet.cc:1722] T 4b30d9c772e2480787ec36513f84f3c5 P 60ba3ffe3b7746469834cd91c84be14e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:47.841200 22500 tablet.cc:1620] T 4b30d9c772e2480787ec36513f84f3c5 P 60ba3ffe3b7746469834cd91c84be14e: MemRowSet was empty: no flush needed.
I20240503 13:56:47.842034 12911 catalog_manager.cc:5576] TS 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153) does not have the latest schema for tablet 4b30d9c772e2480787ec36513f84f3c5 (table rebalancer_dynamic_table [id=0f7ff4c9597044d68254fb58f84de2bb]). Expected version 1 got 0
I20240503 13:56:47.842586 12910 catalog_manager.cc:5576] TS 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695) does not have the latest schema for tablet 4b30d9c772e2480787ec36513f84f3c5 (table rebalancer_dynamic_table [id=0f7ff4c9597044d68254fb58f84de2bb]). Expected version 1 got 0
I20240503 13:56:47.848057 12910 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=0f7ff4c9597044d68254fb58f84de2bb] alter complete (version 1)
I20240503 13:56:47.913296 21492 raft_consensus.cc:482] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:47.913435 21492 raft_consensus.cc:504] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:47.913980 21492 leader_election.cc:288] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:47.914319 15888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289d31dd1f8f40d9b87d7ad401d2d8dd" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7" is_pre_election: true
I20240503 13:56:47.914322 13316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289d31dd1f8f40d9b87d7ad401d2d8dd" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703" is_pre_election: true
I20240503 13:56:47.914443 15888 raft_consensus.cc:2454] T 289d31dd1f8f40d9b87d7ad401d2d8dd P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 0.
I20240503 13:56:47.914467 13316 raft_consensus.cc:2454] T 289d31dd1f8f40d9b87d7ad401d2d8dd P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 0.
I20240503 13:56:47.914743 15426 leader_election.cc:302] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61b887f0d40d497badeca5604c1f0546, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:47.914939 21492 raft_consensus.cc:2781] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:47.915012 21492 raft_consensus.cc:482] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:47.915060 21492 raft_consensus.cc:3037] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:47.915967 21492 raft_consensus.cc:504] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:47.916338 21492 leader_election.cc:288] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 election: Requested vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:47.916651 15888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289d31dd1f8f40d9b87d7ad401d2d8dd" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7"
I20240503 13:56:47.916695 13316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289d31dd1f8f40d9b87d7ad401d2d8dd" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703"
I20240503 13:56:47.916769 15888 raft_consensus.cc:3037] T 289d31dd1f8f40d9b87d7ad401d2d8dd P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:47.916800 13316 raft_consensus.cc:3037] T 289d31dd1f8f40d9b87d7ad401d2d8dd P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:47.917522 15888 raft_consensus.cc:2454] T 289d31dd1f8f40d9b87d7ad401d2d8dd P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 1.
I20240503 13:56:47.917771 13316 raft_consensus.cc:2454] T 289d31dd1f8f40d9b87d7ad401d2d8dd P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 1.
I20240503 13:56:47.917868 15426 leader_election.cc:302] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61b887f0d40d497badeca5604c1f0546, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:47.918152 21492 raft_consensus.cc:2781] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:47.918231 21492 raft_consensus.cc:686] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 [term 1 LEADER]: Becoming Leader. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Running, Role: LEADER
I20240503 13:56:47.918370 21492 consensus_queue.cc:237] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:47.920738 12910 catalog_manager.cc:5484] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61b887f0d40d497badeca5604c1f0546 (127.6.98.69). New cstate: current_term: 1 leader_uuid: "61b887f0d40d497badeca5604c1f0546" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:47.939095 15605 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:56:47.949743 15472 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:56:47.951736 13032 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 13:56:47.953843 12910 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:56:47.955277 13424 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 13:56:47.958400 15867 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:56:47.965036 15738 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:56:47.972705 22498 tablet.cc:1722] T 4b30d9c772e2480787ec36513f84f3c5 P 7690ee79577c45e3a2c8f8b552c0bff8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:47.973052 22498 tablet.cc:1620] T 4b30d9c772e2480787ec36513f84f3c5 P 7690ee79577c45e3a2c8f8b552c0bff8: MemRowSet was empty: no flush needed.
I20240503 13:56:47.973299 22500 tablet.cc:1722] T 4b30d9c772e2480787ec36513f84f3c5 P 60ba3ffe3b7746469834cd91c84be14e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:47.973588 22500 tablet.cc:1620] T 4b30d9c772e2480787ec36513f84f3c5 P 60ba3ffe3b7746469834cd91c84be14e: MemRowSet was empty: no flush needed.
I20240503 13:56:47.975018 22499 tablet.cc:1722] T 4b30d9c772e2480787ec36513f84f3c5 P 61b887f0d40d497badeca5604c1f0546: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:47.975301 22499 tablet.cc:1620] T 4b30d9c772e2480787ec36513f84f3c5 P 61b887f0d40d497badeca5604c1f0546: MemRowSet was empty: no flush needed.
I20240503 13:56:47.978372 13164 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 13:56:47.980114 12911 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=0f7ff4c9597044d68254fb58f84de2bb] alter complete (version 2)
I20240503 13:56:47.990736 13296 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 13:56:48.055946 15757 consensus_queue.cc:237] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:48.057236 13316 raft_consensus.cc:1261] T fcb0aa9fce544ae2a6f1ca96c5315c36 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Refusing update from remote peer 76c386bef61a4bd5a907266a5118ffe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:56:48.057336 15624 raft_consensus.cc:1261] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Refusing update from remote peer 76c386bef61a4bd5a907266a5118ffe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:56:48.057663 21252 consensus_queue.cc:1035] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:56:48.057868 22400 consensus_queue.cc:1035] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:56:48.059794 22400 raft_consensus.cc:2932] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:48.060205 15624 raft_consensus.cc:2932] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:48.060636 13316 raft_consensus.cc:2932] T fcb0aa9fce544ae2a6f1ca96c5315c36 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:48.063889 12910 catalog_manager.cc:5484] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 76c386bef61a4bd5a907266a5118ffe2 reported cstate change: config changed from index 2 to 3, VOTER 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65) evicted. New cstate: current_term: 1 leader_uuid: "76c386bef61a4bd5a907266a5118ffe2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:48.067467 13032 tablet_service.cc:1514] Processing DeleteTablet for tablet fcb0aa9fce544ae2a6f1ca96c5315c36 with delete_type TABLET_DATA_TOMBSTONED (TS 60ba3ffe3b7746469834cd91c84be14e not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:59458
I20240503 13:56:48.068063 22536 tablet_replica.cc:330] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 60ba3ffe3b7746469834cd91c84be14e: stopping tablet replica
I20240503 13:56:48.068298 22536 raft_consensus.cc:2229] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:48.068459 22536 raft_consensus.cc:2258] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:48.068895 22536 ts_tablet_manager.cc:1875] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 60ba3ffe3b7746469834cd91c84be14e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:56:48.071242 22536 ts_tablet_manager.cc:1888] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 60ba3ffe3b7746469834cd91c84be14e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 13:56:48.071310 22536 log.cc:1196] T fcb0aa9fce544ae2a6f1ca96c5315c36 P 60ba3ffe3b7746469834cd91c84be14e: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal/wals/fcb0aa9fce544ae2a6f1ca96c5315c36
I20240503 13:56:48.071887 12897 catalog_manager.cc:4830] TS 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695): tablet fcb0aa9fce544ae2a6f1ca96c5315c36 (table rebalancer_extra_table_6 [id=1161fd279f6d435887536e9c124d251a]) successfully deleted
I20240503 13:56:48.084973 12906 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:56:48.091156 22498 tablet.cc:1722] T 4b30d9c772e2480787ec36513f84f3c5 P 7690ee79577c45e3a2c8f8b552c0bff8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:48.092527 22499 tablet.cc:1722] T 4b30d9c772e2480787ec36513f84f3c5 P 61b887f0d40d497badeca5604c1f0546: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:48.094444 22500 tablet.cc:1722] T 4b30d9c772e2480787ec36513f84f3c5 P 60ba3ffe3b7746469834cd91c84be14e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:48.105654 12906 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table_new_name [id=0f7ff4c9597044d68254fb58f84de2bb] alter complete (version 3)
I20240503 13:56:48.118912 22422 consensus_queue.cc:1035] T e749c1924f504c6f8f780b71865ab0ab P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:48.126000 22422 consensus_queue.cc:1035] T e749c1924f504c6f8f780b71865ab0ab P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:48.192381 21252 consensus_queue.cc:1035] T 867500b772894823b3c240a230c4dfda P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:48.199040 13316 raft_consensus.cc:1203] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:56:48.199617 22400 consensus_queue.cc:1035] T 867500b772894823b3c240a230c4dfda P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:48.211531 12910 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:48.211683 12910 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:48.212433 15605 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:56:48.214586 13296 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 13:56:48.214973 12910 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Sending DeleteTablet for 3 replicas of tablet 4b30d9c772e2480787ec36513f84f3c5
I20240503 13:56:48.215759 13032 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b30d9c772e2480787ec36513f84f3c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:48 UTC) from {username='slave'} at 127.0.0.1:59458
I20240503 13:56:48.215953 13424 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b30d9c772e2480787ec36513f84f3c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:48 UTC) from {username='slave'} at 127.0.0.1:33678
I20240503 13:56:48.216063 22536 tablet_replica.cc:330] T 4b30d9c772e2480787ec36513f84f3c5 P 60ba3ffe3b7746469834cd91c84be14e: stopping tablet replica
I20240503 13:56:48.216217 22536 raft_consensus.cc:2229] T 4b30d9c772e2480787ec36513f84f3c5 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:48.216334 22536 raft_consensus.cc:2258] T 4b30d9c772e2480787ec36513f84f3c5 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:48.216583 15472 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b30d9c772e2480787ec36513f84f3c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:48 UTC) from {username='slave'} at 127.0.0.1:58758
I20240503 13:56:48.217402 22536 ts_tablet_manager.cc:1875] T 4b30d9c772e2480787ec36513f84f3c5 P 60ba3ffe3b7746469834cd91c84be14e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:48.219548 22536 ts_tablet_manager.cc:1888] T 4b30d9c772e2480787ec36513f84f3c5 P 60ba3ffe3b7746469834cd91c84be14e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:48.219609 22536 log.cc:1196] T 4b30d9c772e2480787ec36513f84f3c5 P 60ba3ffe3b7746469834cd91c84be14e: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal/wals/4b30d9c772e2480787ec36513f84f3c5
I20240503 13:56:48.219825 22536 ts_tablet_manager.cc:1909] T 4b30d9c772e2480787ec36513f84f3c5 P 60ba3ffe3b7746469834cd91c84be14e: Deleting consensus metadata
I20240503 13:56:48.221549 22593 tablet_replica.cc:330] T 4b30d9c772e2480787ec36513f84f3c5 P 61b887f0d40d497badeca5604c1f0546: stopping tablet replica
I20240503 13:56:48.221714 22593 raft_consensus.cc:2229] T 4b30d9c772e2480787ec36513f84f3c5 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:48.221835 22593 raft_consensus.cc:2258] T 4b30d9c772e2480787ec36513f84f3c5 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:48.225384 22593 ts_tablet_manager.cc:1875] T 4b30d9c772e2480787ec36513f84f3c5 P 61b887f0d40d497badeca5604c1f0546: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:48.228579 22593 ts_tablet_manager.cc:1888] T 4b30d9c772e2480787ec36513f84f3c5 P 61b887f0d40d497badeca5604c1f0546: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:48.228688 22593 log.cc:1196] T 4b30d9c772e2480787ec36513f84f3c5 P 61b887f0d40d497badeca5604c1f0546: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/wal/wals/4b30d9c772e2480787ec36513f84f3c5
I20240503 13:56:48.229895 22592 tablet_replica.cc:330] T 4b30d9c772e2480787ec36513f84f3c5 P 7690ee79577c45e3a2c8f8b552c0bff8: stopping tablet replica
I20240503 13:56:48.230110 22592 raft_consensus.cc:2229] T 4b30d9c772e2480787ec36513f84f3c5 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:56:48.230357 22592 raft_consensus.cc:2258] T 4b30d9c772e2480787ec36513f84f3c5 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:48.231454 15738 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:56:48.231462 13164 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 13:56:48.231771 22592 ts_tablet_manager.cc:1875] T 4b30d9c772e2480787ec36513f84f3c5 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:48.232422 12897 catalog_manager.cc:4830] TS 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695): tablet 4b30d9c772e2480787ec36513f84f3c5 (table rebalancer_dynamic_table_new_name [id=0f7ff4c9597044d68254fb58f84de2bb]) successfully deleted
I20240503 13:56:48.233325 13424 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:48.233678 15867 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:56:48.236217 22592 ts_tablet_manager.cc:1888] T 4b30d9c772e2480787ec36513f84f3c5 P 7690ee79577c45e3a2c8f8b552c0bff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:48.236351 22592 log.cc:1196] T 4b30d9c772e2480787ec36513f84f3c5 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal/wals/4b30d9c772e2480787ec36513f84f3c5
I20240503 13:56:48.243161 13032 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 13:56:48.245357 15472 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:56:48.247977 22593 ts_tablet_manager.cc:1909] T 4b30d9c772e2480787ec36513f84f3c5 P 61b887f0d40d497badeca5604c1f0546: Deleting consensus metadata
I20240503 13:56:48.253095 12899 catalog_manager.cc:4830] TS 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153): tablet 4b30d9c772e2480787ec36513f84f3c5 (table rebalancer_dynamic_table_new_name [id=0f7ff4c9597044d68254fb58f84de2bb]) successfully deleted
I20240503 13:56:48.266490 22592 ts_tablet_manager.cc:1909] T 4b30d9c772e2480787ec36513f84f3c5 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting consensus metadata
I20240503 13:56:48.267320 12896 catalog_manager.cc:4830] TS 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109): tablet 4b30d9c772e2480787ec36513f84f3c5 (table rebalancer_dynamic_table_new_name [id=0f7ff4c9597044d68254fb58f84de2bb]) successfully deleted
I20240503 13:56:48.312074 13316 consensus_queue.cc:237] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:48.313902 13447 raft_consensus.cc:1261] T 397f0e3e877d4acd9157bdf99bee6255 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:56:48.314316 22577 consensus_queue.cc:1035] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:48.314527 15889 raft_consensus.cc:1261] T 397f0e3e877d4acd9157bdf99bee6255 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:56:48.315133 22577 consensus_queue.cc:1035] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:48.316679 12910 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:56:48.318166 22577 raft_consensus.cc:2932] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:48.321568 15889 raft_consensus.cc:2932] T 397f0e3e877d4acd9157bdf99bee6255 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:48.324050 13424 tablet_service.cc:1467] Processing CreateTablet for tablet 7ad1be0976d34eeb8efb1037707328a6 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a36ebb434ab54293800b8a8852695bf0]), partition=
I20240503 13:56:48.324309 13424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ad1be0976d34eeb8efb1037707328a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:48.324247 13447 raft_consensus.cc:2932] T 397f0e3e877d4acd9157bdf99bee6255 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:48.324332 13164 tablet_service.cc:1467] Processing CreateTablet for tablet 7ad1be0976d34eeb8efb1037707328a6 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a36ebb434ab54293800b8a8852695bf0]), partition=
I20240503 13:56:48.324645 13164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ad1be0976d34eeb8efb1037707328a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:48.324792 15738 tablet_service.cc:1467] Processing CreateTablet for tablet 7ad1be0976d34eeb8efb1037707328a6 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a36ebb434ab54293800b8a8852695bf0]), partition=
I20240503 13:56:48.325011 15738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ad1be0976d34eeb8efb1037707328a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:48.327472 12910 catalog_manager.cc:5484] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 reported cstate change: config changed from index 4 to 5, VOTER 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66) evicted. New cstate: current_term: 1 leader_uuid: "ba0194098aa84449ae17e8a0d4c43703" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:48.328150 22621 tablet_bootstrap.cc:492] T 7ad1be0976d34eeb8efb1037707328a6 P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:48.329170 20722 consensus_queue.cc:1035] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:48.329439 22621 tablet_bootstrap.cc:654] T 7ad1be0976d34eeb8efb1037707328a6 P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:48.330664 22619 tablet_bootstrap.cc:492] T 7ad1be0976d34eeb8efb1037707328a6 P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:48.331665 22621 tablet_bootstrap.cc:492] T 7ad1be0976d34eeb8efb1037707328a6 P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:48.331758 22621 ts_tablet_manager.cc:1367] T 7ad1be0976d34eeb8efb1037707328a6 P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:56:48.332202 22621 raft_consensus.cc:348] T 7ad1be0976d34eeb8efb1037707328a6 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:48.332306 22621 raft_consensus.cc:374] T 7ad1be0976d34eeb8efb1037707328a6 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:48.332336 22621 raft_consensus.cc:729] T 7ad1be0976d34eeb8efb1037707328a6 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:48.332446 22621 consensus_queue.cc:260] T 7ad1be0976d34eeb8efb1037707328a6 P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:48.332562 22619 tablet_bootstrap.cc:654] T 7ad1be0976d34eeb8efb1037707328a6 P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:48.332754 22621 ts_tablet_manager.cc:1398] T 7ad1be0976d34eeb8efb1037707328a6 P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:48.333828 13164 tablet_service.cc:1514] Processing DeleteTablet for tablet 397f0e3e877d4acd9157bdf99bee6255 with delete_type TABLET_DATA_TOMBSTONED (TS 8ccf5a4ac450414c9d3579bd5800b770 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:41668
I20240503 13:56:48.334229 22620 tablet_bootstrap.cc:492] T 7ad1be0976d34eeb8efb1037707328a6 P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:48.337831 22620 tablet_bootstrap.cc:654] T 7ad1be0976d34eeb8efb1037707328a6 P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:48.338507 22625 tablet_replica.cc:330] T 397f0e3e877d4acd9157bdf99bee6255 P 8ccf5a4ac450414c9d3579bd5800b770: stopping tablet replica
I20240503 13:56:48.338696 22625 raft_consensus.cc:2229] T 397f0e3e877d4acd9157bdf99bee6255 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:48.338912 22625 raft_consensus.cc:2258] T 397f0e3e877d4acd9157bdf99bee6255 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:48.339179 20722 consensus_queue.cc:1035] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:48.340649 22625 ts_tablet_manager.cc:1875] T 397f0e3e877d4acd9157bdf99bee6255 P 8ccf5a4ac450414c9d3579bd5800b770: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:56:48.344584 22625 ts_tablet_manager.cc:1888] T 397f0e3e877d4acd9157bdf99bee6255 P 8ccf5a4ac450414c9d3579bd5800b770: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 13:56:48.344771 22625 log.cc:1196] T 397f0e3e877d4acd9157bdf99bee6255 P 8ccf5a4ac450414c9d3579bd5800b770: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/wal/wals/397f0e3e877d4acd9157bdf99bee6255
I20240503 13:56:48.345029 22619 tablet_bootstrap.cc:492] T 7ad1be0976d34eeb8efb1037707328a6 P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:48.345180 22619 ts_tablet_manager.cc:1367] T 7ad1be0976d34eeb8efb1037707328a6 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.002s
I20240503 13:56:48.345707 22619 raft_consensus.cc:348] T 7ad1be0976d34eeb8efb1037707328a6 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:48.345829 12899 catalog_manager.cc:4830] TS 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387): tablet 397f0e3e877d4acd9157bdf99bee6255 (table rebalance_test_table_15 [id=2bf18d88eb674480901ad018abd82ad1]) successfully deleted
I20240503 13:56:48.345896 22619 raft_consensus.cc:374] T 7ad1be0976d34eeb8efb1037707328a6 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:48.345938 22619 raft_consensus.cc:729] T 7ad1be0976d34eeb8efb1037707328a6 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:48.346069 22619 consensus_queue.cc:260] T 7ad1be0976d34eeb8efb1037707328a6 P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:48.346521 22619 ts_tablet_manager.cc:1398] T 7ad1be0976d34eeb8efb1037707328a6 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:48.348410 22620 tablet_bootstrap.cc:492] T 7ad1be0976d34eeb8efb1037707328a6 P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:48.348790 22620 ts_tablet_manager.cc:1367] T 7ad1be0976d34eeb8efb1037707328a6 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240503 13:56:48.349287 22620 raft_consensus.cc:348] T 7ad1be0976d34eeb8efb1037707328a6 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:48.349402 22620 raft_consensus.cc:374] T 7ad1be0976d34eeb8efb1037707328a6 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:48.349435 22620 raft_consensus.cc:729] T 7ad1be0976d34eeb8efb1037707328a6 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:48.349565 22620 consensus_queue.cc:260] T 7ad1be0976d34eeb8efb1037707328a6 P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:48.350582 22620 ts_tablet_manager.cc:1398] T 7ad1be0976d34eeb8efb1037707328a6 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:48.434105 12910 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_extra_table_50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:56:48.446097 13034 tablet_service.cc:1467] Processing CreateTablet for tablet 48f1801af37a472d9b0479122cb1d27e (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=aa04f6a748e64b3f9a62d3501a518118]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:48.446430 13034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48f1801af37a472d9b0479122cb1d27e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:48.446939 13032 tablet_service.cc:1467] Processing CreateTablet for tablet 07750e59b18d40379a70e1d3213306d4 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=aa04f6a748e64b3f9a62d3501a518118]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:48.446940 15604 tablet_service.cc:1467] Processing CreateTablet for tablet a7bf405bf11042c394477c63ea98e3b2 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=aa04f6a748e64b3f9a62d3501a518118]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:48.447186 13032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07750e59b18d40379a70e1d3213306d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:48.447187 15604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7bf405bf11042c394477c63ea98e3b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:48.448804 13424 tablet_service.cc:1467] Processing CreateTablet for tablet 07750e59b18d40379a70e1d3213306d4 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=aa04f6a748e64b3f9a62d3501a518118]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:48.448913 13423 tablet_service.cc:1467] Processing CreateTablet for tablet 48f1801af37a472d9b0479122cb1d27e (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=aa04f6a748e64b3f9a62d3501a518118]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:48.449077 13424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07750e59b18d40379a70e1d3213306d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:48.446940 15605 tablet_service.cc:1467] Processing CreateTablet for tablet 07750e59b18d40379a70e1d3213306d4 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=aa04f6a748e64b3f9a62d3501a518118]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:48.449527 13296 tablet_service.cc:1467] Processing CreateTablet for tablet a7bf405bf11042c394477c63ea98e3b2 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=aa04f6a748e64b3f9a62d3501a518118]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:48.449698 15605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07750e59b18d40379a70e1d3213306d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:48.449780 13296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7bf405bf11042c394477c63ea98e3b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:48.452339 13423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48f1801af37a472d9b0479122cb1d27e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:48.452288 15867 tablet_service.cc:1467] Processing CreateTablet for tablet 48f1801af37a472d9b0479122cb1d27e (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=aa04f6a748e64b3f9a62d3501a518118]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:48.452584 15867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48f1801af37a472d9b0479122cb1d27e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:48.453190 22619 tablet_bootstrap.cc:492] T 07750e59b18d40379a70e1d3213306d4 P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:48.454154 22619 tablet_bootstrap.cc:654] T 07750e59b18d40379a70e1d3213306d4 P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:48.454994 13164 tablet_service.cc:1467] Processing CreateTablet for tablet a7bf405bf11042c394477c63ea98e3b2 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=aa04f6a748e64b3f9a62d3501a518118]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:48.455260 13164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7bf405bf11042c394477c63ea98e3b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:48.457151 22696 tablet_bootstrap.cc:492] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:48.458501 22696 tablet_bootstrap.cc:654] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:48.462008 22620 tablet_bootstrap.cc:492] T a7bf405bf11042c394477c63ea98e3b2 P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:48.462563 22696 tablet_bootstrap.cc:492] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:48.462671 22696 ts_tablet_manager.cc:1367] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 13:56:48.462841 22620 tablet_bootstrap.cc:654] T a7bf405bf11042c394477c63ea98e3b2 P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:48.463335 22696 raft_consensus.cc:348] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:48.463455 22696 raft_consensus.cc:374] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:48.463490 22696 raft_consensus.cc:729] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:48.463702 22696 consensus_queue.cc:260] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:48.465337 22696 ts_tablet_manager.cc:1398] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:56:48.465553 22696 tablet_bootstrap.cc:492] T 07750e59b18d40379a70e1d3213306d4 P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:48.467037 22696 tablet_bootstrap.cc:654] T 07750e59b18d40379a70e1d3213306d4 P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:48.468572 22696 tablet_bootstrap.cc:492] T 07750e59b18d40379a70e1d3213306d4 P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:48.468816 22696 ts_tablet_manager.cc:1367] T 07750e59b18d40379a70e1d3213306d4 P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20240503 13:56:48.469302 22693 tablet_bootstrap.cc:492] T a7bf405bf11042c394477c63ea98e3b2 P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:48.471693 22693 tablet_bootstrap.cc:654] T a7bf405bf11042c394477c63ea98e3b2 P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:48.470571 22696 raft_consensus.cc:348] T 07750e59b18d40379a70e1d3213306d4 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:48.473446 22694 tablet_bootstrap.cc:492] T 48f1801af37a472d9b0479122cb1d27e P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:48.473783 22620 tablet_bootstrap.cc:492] T a7bf405bf11042c394477c63ea98e3b2 P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:48.473896 22620 ts_tablet_manager.cc:1367] T a7bf405bf11042c394477c63ea98e3b2 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 13:56:48.474427 22620 raft_consensus.cc:348] T a7bf405bf11042c394477c63ea98e3b2 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:48.474536 22620 raft_consensus.cc:374] T a7bf405bf11042c394477c63ea98e3b2 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:48.474567 22620 raft_consensus.cc:729] T a7bf405bf11042c394477c63ea98e3b2 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:48.474676 22696 raft_consensus.cc:374] T 07750e59b18d40379a70e1d3213306d4 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:48.474723 22696 raft_consensus.cc:729] T 07750e59b18d40379a70e1d3213306d4 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:48.474684 22620 consensus_queue.cc:260] T a7bf405bf11042c394477c63ea98e3b2 P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:48.474889 22696 consensus_queue.cc:260] T 07750e59b18d40379a70e1d3213306d4 P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:48.474982 22620 ts_tablet_manager.cc:1398] T a7bf405bf11042c394477c63ea98e3b2 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:48.475919 22696 ts_tablet_manager.cc:1398] T 07750e59b18d40379a70e1d3213306d4 P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.007s user 0.000s sys 0.001s
I20240503 13:56:48.479601 22693 tablet_bootstrap.cc:492] T a7bf405bf11042c394477c63ea98e3b2 P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:48.480103 22693 ts_tablet_manager.cc:1367] T a7bf405bf11042c394477c63ea98e3b2 P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 13:56:48.480675 22693 raft_consensus.cc:348] T a7bf405bf11042c394477c63ea98e3b2 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:48.481666 22619 tablet_bootstrap.cc:492] T 07750e59b18d40379a70e1d3213306d4 P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:48.481774 22619 ts_tablet_manager.cc:1367] T 07750e59b18d40379a70e1d3213306d4 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.029s user 0.000s sys 0.002s
I20240503 13:56:48.482131 22693 raft_consensus.cc:374] T a7bf405bf11042c394477c63ea98e3b2 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:48.482182 22693 raft_consensus.cc:729] T a7bf405bf11042c394477c63ea98e3b2 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:48.482251 22619 raft_consensus.cc:348] T 07750e59b18d40379a70e1d3213306d4 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:48.482357 22619 raft_consensus.cc:374] T 07750e59b18d40379a70e1d3213306d4 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:48.482314 22693 consensus_queue.cc:260] T a7bf405bf11042c394477c63ea98e3b2 P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:48.482384 22619 raft_consensus.cc:729] T 07750e59b18d40379a70e1d3213306d4 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:48.482491 22619 consensus_queue.cc:260] T 07750e59b18d40379a70e1d3213306d4 P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:48.482601 22693 ts_tablet_manager.cc:1398] T a7bf405bf11042c394477c63ea98e3b2 P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:48.482759 22619 ts_tablet_manager.cc:1398] T 07750e59b18d40379a70e1d3213306d4 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:48.482944 22619 tablet_bootstrap.cc:492] T 48f1801af37a472d9b0479122cb1d27e P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:48.483067 22692 tablet_bootstrap.cc:492] T a7bf405bf11042c394477c63ea98e3b2 P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:48.478641 22694 tablet_bootstrap.cc:654] T 48f1801af37a472d9b0479122cb1d27e P e47105a0f46e4511b7112e571688a4c7: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:48.484009 22692 tablet_bootstrap.cc:654] T a7bf405bf11042c394477c63ea98e3b2 P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:48.484033 22619 tablet_bootstrap.cc:654] T 48f1801af37a472d9b0479122cb1d27e P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:48.485875 22694 tablet_bootstrap.cc:492] T 48f1801af37a472d9b0479122cb1d27e P e47105a0f46e4511b7112e571688a4c7: No bootstrap required, opened a new log
I20240503 13:56:48.486034 22694 ts_tablet_manager.cc:1367] T 48f1801af37a472d9b0479122cb1d27e P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 13:56:48.486546 22694 raft_consensus.cc:348] T 48f1801af37a472d9b0479122cb1d27e P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:48.486819 22619 tablet_bootstrap.cc:492] T 48f1801af37a472d9b0479122cb1d27e P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:48.486908 22619 ts_tablet_manager.cc:1367] T 48f1801af37a472d9b0479122cb1d27e P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 13:56:48.487061 22694 raft_consensus.cc:374] T 48f1801af37a472d9b0479122cb1d27e P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:48.487112 22694 raft_consensus.cc:729] T 48f1801af37a472d9b0479122cb1d27e P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:48.487296 22694 consensus_queue.cc:260] T 48f1801af37a472d9b0479122cb1d27e P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:48.487326 22619 raft_consensus.cc:348] T 48f1801af37a472d9b0479122cb1d27e P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:48.487417 22619 raft_consensus.cc:374] T 48f1801af37a472d9b0479122cb1d27e P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:48.487444 22619 raft_consensus.cc:729] T 48f1801af37a472d9b0479122cb1d27e P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:48.487557 22619 consensus_queue.cc:260] T 48f1801af37a472d9b0479122cb1d27e P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:48.487797 22619 ts_tablet_manager.cc:1398] T 48f1801af37a472d9b0479122cb1d27e P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:48.488234 22694 ts_tablet_manager.cc:1398] T 48f1801af37a472d9b0479122cb1d27e P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:48.489434 22692 tablet_bootstrap.cc:492] T a7bf405bf11042c394477c63ea98e3b2 P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:48.489548 22692 ts_tablet_manager.cc:1367] T a7bf405bf11042c394477c63ea98e3b2 P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240503 13:56:48.490043 22692 raft_consensus.cc:348] T a7bf405bf11042c394477c63ea98e3b2 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:48.490147 22692 raft_consensus.cc:374] T a7bf405bf11042c394477c63ea98e3b2 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:48.490176 22692 raft_consensus.cc:729] T a7bf405bf11042c394477c63ea98e3b2 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:48.490296 22692 consensus_queue.cc:260] T a7bf405bf11042c394477c63ea98e3b2 P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:48.490581 22692 ts_tablet_manager.cc:1398] T a7bf405bf11042c394477c63ea98e3b2 P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:48.490751 22692 tablet_bootstrap.cc:492] T 07750e59b18d40379a70e1d3213306d4 P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:48.491695 22692 tablet_bootstrap.cc:654] T 07750e59b18d40379a70e1d3213306d4 P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:48.492836 22692 tablet_bootstrap.cc:492] T 07750e59b18d40379a70e1d3213306d4 P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:48.492916 22692 ts_tablet_manager.cc:1367] T 07750e59b18d40379a70e1d3213306d4 P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 13:56:48.493271 22692 raft_consensus.cc:348] T 07750e59b18d40379a70e1d3213306d4 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:48.493357 22692 raft_consensus.cc:374] T 07750e59b18d40379a70e1d3213306d4 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:48.493386 22692 raft_consensus.cc:729] T 07750e59b18d40379a70e1d3213306d4 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:48.493525 22692 consensus_queue.cc:260] T 07750e59b18d40379a70e1d3213306d4 P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:48.494077 22692 ts_tablet_manager.cc:1398] T 07750e59b18d40379a70e1d3213306d4 P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:48.523020 20870 raft_consensus.cc:482] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:48.523177 20870 raft_consensus.cc:504] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:48.523581 20870 leader_election.cc:288] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:48.524106 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f1801af37a472d9b0479122cb1d27e" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
I20240503 13:56:48.524121 15889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f1801af37a472d9b0479122cb1d27e" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7" is_pre_election: true
I20240503 13:56:48.524271 13447 raft_consensus.cc:2454] T 48f1801af37a472d9b0479122cb1d27e P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 0.
I20240503 13:56:48.524282 15889 raft_consensus.cc:2454] T 48f1801af37a472d9b0479122cb1d27e P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 0.
I20240503 13:56:48.524569 12987 leader_election.cc:302] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 7690ee79577c45e3a2c8f8b552c0bff8; no voters:
I20240503 13:56:48.524729 20870 raft_consensus.cc:2781] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:48.524813 20870 raft_consensus.cc:482] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:48.524855 20870 raft_consensus.cc:3037] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:48.525863 20870 raft_consensus.cc:504] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:48.526204 20870 leader_election.cc:288] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 election: Requested vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:48.526609 15889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f1801af37a472d9b0479122cb1d27e" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7"
I20240503 13:56:48.526743 15889 raft_consensus.cc:3037] T 48f1801af37a472d9b0479122cb1d27e P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:48.527653 15889 raft_consensus.cc:2454] T 48f1801af37a472d9b0479122cb1d27e P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 1.
I20240503 13:56:48.527884 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f1801af37a472d9b0479122cb1d27e" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
I20240503 13:56:48.527920 12987 leader_election.cc:302] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:48.528005 13447 raft_consensus.cc:3037] T 48f1801af37a472d9b0479122cb1d27e P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:48.528095 20870 raft_consensus.cc:2781] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:48.528231 20870 raft_consensus.cc:686] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e [term 1 LEADER]: Becoming Leader. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Running, Role: LEADER
I20240503 13:56:48.528439 20870 consensus_queue.cc:237] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:48.528963 13447 raft_consensus.cc:2454] T 48f1801af37a472d9b0479122cb1d27e P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 1.
I20240503 13:56:48.531287 12906 catalog_manager.cc:5484] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e reported cstate change: term changed from 0 to 1, leader changed from <none> to 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65). New cstate: current_term: 1 leader_uuid: "60ba3ffe3b7746469834cd91c84be14e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:48.622805 22400 raft_consensus.cc:482] T 7ad1be0976d34eeb8efb1037707328a6 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:48.622977 22400 raft_consensus.cc:504] T 7ad1be0976d34eeb8efb1037707328a6 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:48.623498 22400 leader_election.cc:288] T 7ad1be0976d34eeb8efb1037707328a6 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:48.624086 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ad1be0976d34eeb8efb1037707328a6" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
I20240503 13:56:48.624140 13183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ad1be0976d34eeb8efb1037707328a6" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770" is_pre_election: true
I20240503 13:56:48.624228 13447 raft_consensus.cc:2454] T 7ad1be0976d34eeb8efb1037707328a6 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 0.
I20240503 13:56:48.624274 13183 raft_consensus.cc:2454] T 7ad1be0976d34eeb8efb1037707328a6 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 0.
I20240503 13:56:48.624522 15691 leader_election.cc:302] T 7ad1be0976d34eeb8efb1037707328a6 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, 76c386bef61a4bd5a907266a5118ffe2; no voters:
I20240503 13:56:48.624889 22400 raft_consensus.cc:2781] T 7ad1be0976d34eeb8efb1037707328a6 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:48.625147 22400 raft_consensus.cc:482] T 7ad1be0976d34eeb8efb1037707328a6 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:48.625203 22400 raft_consensus.cc:3037] T 7ad1be0976d34eeb8efb1037707328a6 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:48.628715 22400 raft_consensus.cc:504] T 7ad1be0976d34eeb8efb1037707328a6 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:48.629225 22400 leader_election.cc:288] T 7ad1be0976d34eeb8efb1037707328a6 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 election: Requested vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:48.629534 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ad1be0976d34eeb8efb1037707328a6" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
I20240503 13:56:48.629554 13183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ad1be0976d34eeb8efb1037707328a6" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770"
I20240503 13:56:48.629650 13447 raft_consensus.cc:3037] T 7ad1be0976d34eeb8efb1037707328a6 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:48.629673 13183 raft_consensus.cc:3037] T 7ad1be0976d34eeb8efb1037707328a6 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:48.630522 13447 raft_consensus.cc:2454] T 7ad1be0976d34eeb8efb1037707328a6 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 1.
I20240503 13:56:48.630666 13183 raft_consensus.cc:2454] T 7ad1be0976d34eeb8efb1037707328a6 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 1.
I20240503 13:56:48.630815 15691 leader_election.cc:302] T 7ad1be0976d34eeb8efb1037707328a6 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, 76c386bef61a4bd5a907266a5118ffe2; no voters:
I20240503 13:56:48.630977 22400 raft_consensus.cc:2781] T 7ad1be0976d34eeb8efb1037707328a6 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:48.631052 22400 raft_consensus.cc:686] T 7ad1be0976d34eeb8efb1037707328a6 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 LEADER]: Becoming Leader. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Running, Role: LEADER
I20240503 13:56:48.631192 22400 consensus_queue.cc:237] T 7ad1be0976d34eeb8efb1037707328a6 P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:48.634037 12906 catalog_manager.cc:5484] T 7ad1be0976d34eeb8efb1037707328a6 P 76c386bef61a4bd5a907266a5118ffe2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71). New cstate: current_term: 1 leader_uuid: "76c386bef61a4bd5a907266a5118ffe2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:48.659751 13032 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 13:56:48.662279 15605 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:56:48.667322 13164 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 13:56:48.679522 15738 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 13:56:48.680094 15867 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:56:48.680550 13423 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:48.683056 13296 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 13:56:48.685546 15472 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:56:48.754137 13051 consensus_queue.cc:237] T 3f5d536cffe742e69f4b0d3b93caf76c P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:48.755471 13316 raft_consensus.cc:1261] T 3f5d536cffe742e69f4b0d3b93caf76c P ba0194098aa84449ae17e8a0d4c43703 [term 2 FOLLOWER]: Refusing update from remote peer 60ba3ffe3b7746469834cd91c84be14e: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 13:56:48.755548 15757 raft_consensus.cc:1261] T 3f5d536cffe742e69f4b0d3b93caf76c P 76c386bef61a4bd5a907266a5118ffe2 [term 2 FOLLOWER]: Refusing update from remote peer 60ba3ffe3b7746469834cd91c84be14e: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 13:56:48.755900 20870 consensus_queue.cc:1035] T 3f5d536cffe742e69f4b0d3b93caf76c P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:48.756084 22705 consensus_queue.cc:1035] T 3f5d536cffe742e69f4b0d3b93caf76c P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:48.758199 22738 raft_consensus.cc:2932] T 3f5d536cffe742e69f4b0d3b93caf76c P 60ba3ffe3b7746469834cd91c84be14e [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 009f849ee7b54c80a3f5264695047e4d (127.6.98.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:48.758702 15757 raft_consensus.cc:2932] T 3f5d536cffe742e69f4b0d3b93caf76c P 76c386bef61a4bd5a907266a5118ffe2 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 009f849ee7b54c80a3f5264695047e4d (127.6.98.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:48.758692 13316 raft_consensus.cc:2932] T 3f5d536cffe742e69f4b0d3b93caf76c P ba0194098aa84449ae17e8a0d4c43703 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 009f849ee7b54c80a3f5264695047e4d (127.6.98.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:48.762192 12905 catalog_manager.cc:5484] T 3f5d536cffe742e69f4b0d3b93caf76c P ba0194098aa84449ae17e8a0d4c43703 reported cstate change: config changed from index 2 to 4, VOTER 009f849ee7b54c80a3f5264695047e4d (127.6.98.70) evicted. New cstate: current_term: 2 leader_uuid: "60ba3ffe3b7746469834cd91c84be14e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:48.765338 15605 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f5d536cffe742e69f4b0d3b93caf76c with delete_type TABLET_DATA_TOMBSTONED (TS 009f849ee7b54c80a3f5264695047e4d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:60718
I20240503 13:56:48.766084 22743 tablet_replica.cc:330] T 3f5d536cffe742e69f4b0d3b93caf76c P 009f849ee7b54c80a3f5264695047e4d: stopping tablet replica
I20240503 13:56:48.766302 22743 raft_consensus.cc:2229] T 3f5d536cffe742e69f4b0d3b93caf76c P 009f849ee7b54c80a3f5264695047e4d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:48.766492 22743 raft_consensus.cc:2258] T 3f5d536cffe742e69f4b0d3b93caf76c P 009f849ee7b54c80a3f5264695047e4d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:48.766973 22743 ts_tablet_manager.cc:1875] T 3f5d536cffe742e69f4b0d3b93caf76c P 009f849ee7b54c80a3f5264695047e4d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:56:48.769783 22743 ts_tablet_manager.cc:1888] T 3f5d536cffe742e69f4b0d3b93caf76c P 009f849ee7b54c80a3f5264695047e4d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240503 13:56:48.769892 22743 log.cc:1196] T 3f5d536cffe742e69f4b0d3b93caf76c P 009f849ee7b54c80a3f5264695047e4d: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/wal/wals/3f5d536cffe742e69f4b0d3b93caf76c
I20240503 13:56:48.770469 12896 catalog_manager.cc:4830] TS 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951): tablet 3f5d536cffe742e69f4b0d3b93caf76c (table rebalancer_extra_table_12 [id=46a3f42046eb4cb89a40f991a54af62b]) successfully deleted
I20240503 13:56:48.798439 12906 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:56:48.798452 13445 consensus_queue.cc:237] T b34b432bb82e465f87a6a3b05d7adfd2 P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:48.800016 13316 raft_consensus.cc:1261] T b34b432bb82e465f87a6a3b05d7adfd2 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Refusing update from remote peer 7690ee79577c45e3a2c8f8b552c0bff8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 13:56:48.800232 13383 consensus_peers.cc:485] T b34b432bb82e465f87a6a3b05d7adfd2 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153): Couldn't send request to peer 61b887f0d40d497badeca5604c1f0546. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b34b432bb82e465f87a6a3b05d7adfd2. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:48.800505 13051 raft_consensus.cc:1261] T b34b432bb82e465f87a6a3b05d7adfd2 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Refusing update from remote peer 7690ee79577c45e3a2c8f8b552c0bff8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:56:48.800441 22762 consensus_queue.cc:1035] T b34b432bb82e465f87a6a3b05d7adfd2 P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:48.801074 22745 consensus_queue.cc:1035] T b34b432bb82e465f87a6a3b05d7adfd2 P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:48.803205 13183 raft_consensus.cc:1261] T 7ad1be0976d34eeb8efb1037707328a6 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Refusing update from remote peer 76c386bef61a4bd5a907266a5118ffe2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:48.803364 13445 raft_consensus.cc:1261] T 7ad1be0976d34eeb8efb1037707328a6 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer 76c386bef61a4bd5a907266a5118ffe2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:48.803781 22400 consensus_queue.cc:1035] T 7ad1be0976d34eeb8efb1037707328a6 P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:48.804476 21252 consensus_queue.cc:1035] T 7ad1be0976d34eeb8efb1037707328a6 P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:48.808246 22745 raft_consensus.cc:2932] T b34b432bb82e465f87a6a3b05d7adfd2 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 61b887f0d40d497badeca5604c1f0546 (127.6.98.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:48.808625 20870 raft_consensus.cc:482] T 07750e59b18d40379a70e1d3213306d4 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:48.808707 20870 raft_consensus.cc:504] T 07750e59b18d40379a70e1d3213306d4 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:48.808691 13314 raft_consensus.cc:2932] T b34b432bb82e465f87a6a3b05d7adfd2 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 61b887f0d40d497badeca5604c1f0546 (127.6.98.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:48.809062 20870 leader_election.cc:288] T 07750e59b18d40379a70e1d3213306d4 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951)
I20240503 13:56:48.809491 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07750e59b18d40379a70e1d3213306d4" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
I20240503 13:56:48.809554 12905 catalog_manager.cc:5576] TS 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109) does not have the latest schema for tablet 7ad1be0976d34eeb8efb1037707328a6 (table rebalancer_dynamic_table [id=a36ebb434ab54293800b8a8852695bf0]). Expected version 1 got 0
I20240503 13:56:48.809613 13447 raft_consensus.cc:2454] T 07750e59b18d40379a70e1d3213306d4 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 0.
I20240503 13:56:48.810537 13051 raft_consensus.cc:2932] T b34b432bb82e465f87a6a3b05d7adfd2 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 61b887f0d40d497badeca5604c1f0546 (127.6.98.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:48.810878 15623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07750e59b18d40379a70e1d3213306d4" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d" is_pre_election: true
I20240503 13:56:48.811007 15623 raft_consensus.cc:2454] T 07750e59b18d40379a70e1d3213306d4 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 0.
I20240503 13:56:48.811563 12987 leader_election.cc:302] T 07750e59b18d40379a70e1d3213306d4 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 60ba3ffe3b7746469834cd91c84be14e; no voters:
I20240503 13:56:48.811839 22738 raft_consensus.cc:2781] T 07750e59b18d40379a70e1d3213306d4 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:48.811910 22738 raft_consensus.cc:482] T 07750e59b18d40379a70e1d3213306d4 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:48.811954 22738 raft_consensus.cc:3037] T 07750e59b18d40379a70e1d3213306d4 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:48.812952 22738 raft_consensus.cc:504] T 07750e59b18d40379a70e1d3213306d4 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:48.813294 22738 leader_election.cc:288] T 07750e59b18d40379a70e1d3213306d4 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 election: Requested vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951)
I20240503 13:56:48.813652 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07750e59b18d40379a70e1d3213306d4" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
I20240503 13:56:48.813757 13447 raft_consensus.cc:3037] T 07750e59b18d40379a70e1d3213306d4 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:48.814707 13447 raft_consensus.cc:2454] T 07750e59b18d40379a70e1d3213306d4 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 1.
I20240503 13:56:48.814925 22771 tablet.cc:1722] T 7ad1be0976d34eeb8efb1037707328a6 P 76c386bef61a4bd5a907266a5118ffe2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:48.815001 12911 catalog_manager.cc:5484] T b34b432bb82e465f87a6a3b05d7adfd2 P 60ba3ffe3b7746469834cd91c84be14e reported cstate change: config changed from index -1 to 4, VOTER 61b887f0d40d497badeca5604c1f0546 (127.6.98.69) added. New cstate: current_term: 1 leader_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:48.815704 22771 tablet.cc:1620] T 7ad1be0976d34eeb8efb1037707328a6 P 76c386bef61a4bd5a907266a5118ffe2: MemRowSet was empty: no flush needed.
I20240503 13:56:48.816181 12906 catalog_manager.cc:5576] TS 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387) does not have the latest schema for tablet 7ad1be0976d34eeb8efb1037707328a6 (table rebalancer_dynamic_table [id=a36ebb434ab54293800b8a8852695bf0]). Expected version 1 got 0
I20240503 13:56:48.816339 12987 leader_election.cc:302] T 07750e59b18d40379a70e1d3213306d4 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 7690ee79577c45e3a2c8f8b552c0bff8; no voters:
I20240503 13:56:48.816537 20870 raft_consensus.cc:2781] T 07750e59b18d40379a70e1d3213306d4 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:48.816618 20870 raft_consensus.cc:686] T 07750e59b18d40379a70e1d3213306d4 P 60ba3ffe3b7746469834cd91c84be14e [term 1 LEADER]: Becoming Leader. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Running, Role: LEADER
I20240503 13:56:48.816738 20870 consensus_queue.cc:237] T 07750e59b18d40379a70e1d3213306d4 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:48.816841 15623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07750e59b18d40379a70e1d3213306d4" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d"
I20240503 13:56:48.816964 15623 raft_consensus.cc:3037] T 07750e59b18d40379a70e1d3213306d4 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:48.818042 15623 raft_consensus.cc:2454] T 07750e59b18d40379a70e1d3213306d4 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 1.
I20240503 13:56:48.818928 22772 tablet.cc:1722] T 7ad1be0976d34eeb8efb1037707328a6 P 8ccf5a4ac450414c9d3579bd5800b770: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:48.819442 22772 tablet.cc:1620] T 7ad1be0976d34eeb8efb1037707328a6 P 8ccf5a4ac450414c9d3579bd5800b770: MemRowSet was empty: no flush needed.
I20240503 13:56:48.819989 22768 mvcc.cc:205] Tried to move back new op lower bound from 7023593917654528000 to 7023593916954906624. Current Snapshot: MvccSnapshot[applied={T|T < 7023593917654528000}]
I20240503 13:56:48.820936 12906 catalog_manager.cc:5576] TS 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387) does not have the latest schema for tablet 7ad1be0976d34eeb8efb1037707328a6 (table rebalancer_dynamic_table [id=a36ebb434ab54293800b8a8852695bf0]). Expected version 1 got 0
I20240503 13:56:48.823606 12905 catalog_manager.cc:5484] T 07750e59b18d40379a70e1d3213306d4 P 60ba3ffe3b7746469834cd91c84be14e reported cstate change: term changed from 0 to 1, leader changed from <none> to 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65). New cstate: current_term: 1 leader_uuid: "60ba3ffe3b7746469834cd91c84be14e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } }
I20240503 13:56:48.824230 12906 catalog_manager.cc:5576] TS 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109) does not have the latest schema for tablet 7ad1be0976d34eeb8efb1037707328a6 (table rebalancer_dynamic_table [id=a36ebb434ab54293800b8a8852695bf0]). Expected version 1 got 0
I20240503 13:56:48.827111 22774 tablet.cc:1722] T 7ad1be0976d34eeb8efb1037707328a6 P 7690ee79577c45e3a2c8f8b552c0bff8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:48.827401 22774 tablet.cc:1620] T 7ad1be0976d34eeb8efb1037707328a6 P 7690ee79577c45e3a2c8f8b552c0bff8: MemRowSet was empty: no flush needed.
I20240503 13:56:48.830000 12910 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=a36ebb434ab54293800b8a8852695bf0] alter complete (version 1)
I20240503 13:56:48.849601 15493 consensus_queue.cc:237] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:48.851100 15623 raft_consensus.cc:1261] T c7d5e8c09ec14ac7891c8a63cb896878 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Refusing update from remote peer 61b887f0d40d497badeca5604c1f0546: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:48.851101 13445 raft_consensus.cc:1261] T c7d5e8c09ec14ac7891c8a63cb896878 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer 61b887f0d40d497badeca5604c1f0546: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 13:56:48.851325 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c7d5e8c09ec14ac7891c8a63cb896878. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:48.851540 22791 consensus_queue.cc:1035] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:56:48.851732 20722 consensus_queue.cc:1035] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:56:48.853446 22791 raft_consensus.cc:2932] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:48.855367 13445 raft_consensus.cc:2932] T c7d5e8c09ec14ac7891c8a63cb896878 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:48.855486 15623 raft_consensus.cc:2932] T c7d5e8c09ec14ac7891c8a63cb896878 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:48.858453 12906 catalog_manager.cc:5484] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 reported cstate change: config changed from index -1 to 2, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) added. New cstate: current_term: 1 leader_uuid: "61b887f0d40d497badeca5604c1f0546" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:48.879895 15757 consensus_queue.cc:237] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:48.881732 15493 raft_consensus.cc:1261] T 604875eff3594876af54e9126057bd7f P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Refusing update from remote peer 76c386bef61a4bd5a907266a5118ffe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:48.881788 15623 raft_consensus.cc:1261] T 604875eff3594876af54e9126057bd7f P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Refusing update from remote peer 76c386bef61a4bd5a907266a5118ffe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 13:56:48.882011 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 604875eff3594876af54e9126057bd7f. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:48.882182 21252 consensus_queue.cc:1035] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:56:48.882726 21252 consensus_queue.cc:1035] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:56:48.884562 22400 raft_consensus.cc:2932] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:48.884924 15623 raft_consensus.cc:2932] T 604875eff3594876af54e9126057bd7f P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:48.886775 15493 raft_consensus.cc:2932] T 604875eff3594876af54e9126057bd7f P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:48.888227 12905 catalog_manager.cc:5484] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 reported cstate change: config changed from index -1 to 2, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) added. New cstate: current_term: 1 leader_uuid: "76c386bef61a4bd5a907266a5118ffe2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:48.908001 22614 raft_consensus.cc:482] T a7bf405bf11042c394477c63ea98e3b2 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:48.908140 22614 raft_consensus.cc:504] T a7bf405bf11042c394477c63ea98e3b2 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:48.908838 22614 leader_election.cc:288] T a7bf405bf11042c394477c63ea98e3b2 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:48.909241 13183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7bf405bf11042c394477c63ea98e3b2" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770" is_pre_election: true
I20240503 13:56:48.909399 13183 raft_consensus.cc:2454] T a7bf405bf11042c394477c63ea98e3b2 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 0.
I20240503 13:56:48.909771 13252 leader_election.cc:302] T a7bf405bf11042c394477c63ea98e3b2 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ccf5a4ac450414c9d3579bd5800b770, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:48.909914 15623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7bf405bf11042c394477c63ea98e3b2" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d" is_pre_election: true
I20240503 13:56:48.909976 22614 raft_consensus.cc:2781] T a7bf405bf11042c394477c63ea98e3b2 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:48.910028 22614 raft_consensus.cc:482] T a7bf405bf11042c394477c63ea98e3b2 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:48.910061 22614 raft_consensus.cc:3037] T a7bf405bf11042c394477c63ea98e3b2 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:48.910079 15623 raft_consensus.cc:2454] T a7bf405bf11042c394477c63ea98e3b2 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 0.
I20240503 13:56:48.911160 22614 raft_consensus.cc:504] T a7bf405bf11042c394477c63ea98e3b2 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:48.911500 22614 leader_election.cc:288] T a7bf405bf11042c394477c63ea98e3b2 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: Requested vote from peers 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:48.911804 15623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7bf405bf11042c394477c63ea98e3b2" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d"
I20240503 13:56:48.911929 15623 raft_consensus.cc:3037] T a7bf405bf11042c394477c63ea98e3b2 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:48.912792 13183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7bf405bf11042c394477c63ea98e3b2" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770"
I20240503 13:56:48.912922 13183 raft_consensus.cc:3037] T a7bf405bf11042c394477c63ea98e3b2 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:48.913172 15623 raft_consensus.cc:2454] T a7bf405bf11042c394477c63ea98e3b2 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 1.
I20240503 13:56:48.913483 13249 leader_election.cc:302] T a7bf405bf11042c394477c63ea98e3b2 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:48.913645 22577 raft_consensus.cc:2781] T a7bf405bf11042c394477c63ea98e3b2 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:48.913805 13183 raft_consensus.cc:2454] T a7bf405bf11042c394477c63ea98e3b2 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 1.
I20240503 13:56:48.913817 22577 raft_consensus.cc:686] T a7bf405bf11042c394477c63ea98e3b2 P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Becoming Leader. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Running, Role: LEADER
I20240503 13:56:48.913955 22577 consensus_queue.cc:237] T a7bf405bf11042c394477c63ea98e3b2 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:48.916378 12906 catalog_manager.cc:5484] T a7bf405bf11042c394477c63ea98e3b2 P ba0194098aa84449ae17e8a0d4c43703 reported cstate change: term changed from 0 to 1, leader changed from <none> to ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67). New cstate: current_term: 1 leader_uuid: "ba0194098aa84449ae17e8a0d4c43703" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } }
I20240503 13:56:48.931452 12910 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:56:48.938089 22771 tablet.cc:1722] T 7ad1be0976d34eeb8efb1037707328a6 P 76c386bef61a4bd5a907266a5118ffe2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:48.938462 22771 tablet.cc:1620] T 7ad1be0976d34eeb8efb1037707328a6 P 76c386bef61a4bd5a907266a5118ffe2: MemRowSet was empty: no flush needed.
I20240503 13:56:48.938618 22772 tablet.cc:1722] T 7ad1be0976d34eeb8efb1037707328a6 P 8ccf5a4ac450414c9d3579bd5800b770: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:48.938906 22772 tablet.cc:1620] T 7ad1be0976d34eeb8efb1037707328a6 P 8ccf5a4ac450414c9d3579bd5800b770: MemRowSet was empty: no flush needed.
I20240503 13:56:48.939159 22774 tablet.cc:1722] T 7ad1be0976d34eeb8efb1037707328a6 P 7690ee79577c45e3a2c8f8b552c0bff8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:48.939491 22774 tablet.cc:1620] T 7ad1be0976d34eeb8efb1037707328a6 P 7690ee79577c45e3a2c8f8b552c0bff8: MemRowSet was empty: no flush needed.
I20240503 13:56:48.943161 22738 consensus_queue.cc:1035] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:48.945003 12906 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=a36ebb434ab54293800b8a8852695bf0] alter complete (version 2)
I20240503 13:56:48.952179 22738 consensus_queue.cc:1035] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:49.018011 15738 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 13:56:49.018285 15472 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:56:49.018352 13032 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 13:56:49.024672 15605 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:56:49.035032 13296 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 13:56:49.041131 13423 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:49.048341 12908 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:56:49.057469 22771 tablet.cc:1722] T 7ad1be0976d34eeb8efb1037707328a6 P 76c386bef61a4bd5a907266a5118ffe2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:49.061132 13164 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 13:56:49.061889 22772 tablet.cc:1722] T 7ad1be0976d34eeb8efb1037707328a6 P 8ccf5a4ac450414c9d3579bd5800b770: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:49.065935 22774 tablet.cc:1722] T 7ad1be0976d34eeb8efb1037707328a6 P 7690ee79577c45e3a2c8f8b552c0bff8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:49.066264 15867 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:56:49.071537 12908 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table_new_name [id=a36ebb434ab54293800b8a8852695bf0] alter complete (version 3)
I20240503 13:56:49.185286 12912 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:49.185544 12912 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:49.188220 12912 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Sending DeleteTablet for 3 replicas of tablet 7ad1be0976d34eeb8efb1037707328a6
I20240503 13:56:49.188874 13423 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ad1be0976d34eeb8efb1037707328a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:49 UTC) from {username='slave'} at 127.0.0.1:33678
I20240503 13:56:49.188973 13164 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ad1be0976d34eeb8efb1037707328a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:49 UTC) from {username='slave'} at 127.0.0.1:41668
I20240503 13:56:49.189337 15738 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ad1be0976d34eeb8efb1037707328a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:49 UTC) from {username='slave'} at 127.0.0.1:44984
I20240503 13:56:49.190469 22913 tablet_replica.cc:330] T 7ad1be0976d34eeb8efb1037707328a6 P 76c386bef61a4bd5a907266a5118ffe2: stopping tablet replica
I20240503 13:56:49.190668 22913 raft_consensus.cc:2229] T 7ad1be0976d34eeb8efb1037707328a6 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:56:49.190925 22913 raft_consensus.cc:2258] T 7ad1be0976d34eeb8efb1037707328a6 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:49.191455 22913 ts_tablet_manager.cc:1875] T 7ad1be0976d34eeb8efb1037707328a6 P 76c386bef61a4bd5a907266a5118ffe2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:49.198279 22912 tablet_replica.cc:330] T 7ad1be0976d34eeb8efb1037707328a6 P 8ccf5a4ac450414c9d3579bd5800b770: stopping tablet replica
I20240503 13:56:49.198531 22912 raft_consensus.cc:2229] T 7ad1be0976d34eeb8efb1037707328a6 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:49.198731 22912 raft_consensus.cc:2258] T 7ad1be0976d34eeb8efb1037707328a6 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:49.199055 22911 tablet_replica.cc:330] T 7ad1be0976d34eeb8efb1037707328a6 P 7690ee79577c45e3a2c8f8b552c0bff8: stopping tablet replica
I20240503 13:56:49.199203 22911 raft_consensus.cc:2229] T 7ad1be0976d34eeb8efb1037707328a6 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:49.199321 22911 raft_consensus.cc:2258] T 7ad1be0976d34eeb8efb1037707328a6 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:49.199801 22912 ts_tablet_manager.cc:1875] T 7ad1be0976d34eeb8efb1037707328a6 P 8ccf5a4ac450414c9d3579bd5800b770: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:49.200999 22913 ts_tablet_manager.cc:1888] T 7ad1be0976d34eeb8efb1037707328a6 P 76c386bef61a4bd5a907266a5118ffe2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:49.201066 22913 log.cc:1196] T 7ad1be0976d34eeb8efb1037707328a6 P 76c386bef61a4bd5a907266a5118ffe2: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/wal/wals/7ad1be0976d34eeb8efb1037707328a6
I20240503 13:56:49.201051 22911 ts_tablet_manager.cc:1875] T 7ad1be0976d34eeb8efb1037707328a6 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:49.201269 22913 ts_tablet_manager.cc:1909] T 7ad1be0976d34eeb8efb1037707328a6 P 76c386bef61a4bd5a907266a5118ffe2: Deleting consensus metadata
I20240503 13:56:49.203685 12899 catalog_manager.cc:4830] TS 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885): tablet 7ad1be0976d34eeb8efb1037707328a6 (table rebalancer_dynamic_table_new_name [id=a36ebb434ab54293800b8a8852695bf0]) successfully deleted
I20240503 13:56:49.203982 22911 ts_tablet_manager.cc:1888] T 7ad1be0976d34eeb8efb1037707328a6 P 7690ee79577c45e3a2c8f8b552c0bff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:49.204077 22911 log.cc:1196] T 7ad1be0976d34eeb8efb1037707328a6 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal/wals/7ad1be0976d34eeb8efb1037707328a6
I20240503 13:56:49.204397 22911 ts_tablet_manager.cc:1909] T 7ad1be0976d34eeb8efb1037707328a6 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting consensus metadata
I20240503 13:56:49.205267 12896 catalog_manager.cc:4830] TS 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109): tablet 7ad1be0976d34eeb8efb1037707328a6 (table rebalancer_dynamic_table_new_name [id=a36ebb434ab54293800b8a8852695bf0]) successfully deleted
I20240503 13:56:49.206060 22912 ts_tablet_manager.cc:1888] T 7ad1be0976d34eeb8efb1037707328a6 P 8ccf5a4ac450414c9d3579bd5800b770: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:49.206153 22912 log.cc:1196] T 7ad1be0976d34eeb8efb1037707328a6 P 8ccf5a4ac450414c9d3579bd5800b770: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/wal/wals/7ad1be0976d34eeb8efb1037707328a6
I20240503 13:56:49.207530 22914 ts_tablet_manager.cc:902] T b34b432bb82e465f87a6a3b05d7adfd2 P 61b887f0d40d497badeca5604c1f0546: Initiating tablet copy from peer 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:49.208045 22914 tablet_copy_client.cc:308] T b34b432bb82e465f87a6a3b05d7adfd2 P 61b887f0d40d497badeca5604c1f0546: tablet copy: Beginning tablet copy session from remote peer at address 127.6.98.68:36109
I20240503 13:56:49.208475 13467 tablet_copy_service.cc:140] P 7690ee79577c45e3a2c8f8b552c0bff8: Received BeginTabletCopySession request for tablet b34b432bb82e465f87a6a3b05d7adfd2 from peer 61b887f0d40d497badeca5604c1f0546 ({username='slave'} at 127.6.98.69:40065)
I20240503 13:56:49.208599 13467 tablet_copy_service.cc:161] P 7690ee79577c45e3a2c8f8b552c0bff8: Beginning new tablet copy session on tablet b34b432bb82e465f87a6a3b05d7adfd2 from peer 61b887f0d40d497badeca5604c1f0546 at {username='slave'} at 127.6.98.69:40065: session id = 61b887f0d40d497badeca5604c1f0546-b34b432bb82e465f87a6a3b05d7adfd2
I20240503 13:56:49.209429 13467 tablet_copy_source_session.cc:205] T b34b432bb82e465f87a6a3b05d7adfd2 P 7690ee79577c45e3a2c8f8b552c0bff8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:56:49.210444 22912 ts_tablet_manager.cc:1909] T 7ad1be0976d34eeb8efb1037707328a6 P 8ccf5a4ac450414c9d3579bd5800b770: Deleting consensus metadata
I20240503 13:56:49.210539 22914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b34b432bb82e465f87a6a3b05d7adfd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:49.213411 22914 tablet_copy_client.cc:783] T b34b432bb82e465f87a6a3b05d7adfd2 P 61b887f0d40d497badeca5604c1f0546: tablet copy: Starting download of 0 data blocks...
I20240503 13:56:49.213641 22914 tablet_copy_client.cc:647] T b34b432bb82e465f87a6a3b05d7adfd2 P 61b887f0d40d497badeca5604c1f0546: tablet copy: Starting download of 1 WAL segments...
I20240503 13:56:49.215889 12899 catalog_manager.cc:4830] TS 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387): tablet 7ad1be0976d34eeb8efb1037707328a6 (table rebalancer_dynamic_table_new_name [id=a36ebb434ab54293800b8a8852695bf0]) successfully deleted
I20240503 13:56:49.216405 22914 tablet_copy_client.cc:523] T b34b432bb82e465f87a6a3b05d7adfd2 P 61b887f0d40d497badeca5604c1f0546: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:56:49.218124 22914 tablet_bootstrap.cc:492] T b34b432bb82e465f87a6a3b05d7adfd2 P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:49.228014 15605 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:56:49.229085 15472 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:56:49.234097 15738 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:56:49.234241 20870 consensus_queue.cc:1035] T 07750e59b18d40379a70e1d3213306d4 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:49.236752 13032 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 13:56:49.241044 13423 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:49.248833 13296 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 13:56:49.256426 15867 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:56:49.268915 20870 consensus_queue.cc:1035] T 07750e59b18d40379a70e1d3213306d4 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:49.269599 13164 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 13:56:49.291014 12905 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:56:49.298931 15472 tablet_service.cc:1467] Processing CreateTablet for tablet 6be43f8163a043f199d87ec2f969745a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=53f10ccab76d45c3a497fed6e684bef5]), partition=
I20240503 13:56:49.299253 15472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6be43f8163a043f199d87ec2f969745a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:49.301909 22914 tablet_bootstrap.cc:492] T b34b432bb82e465f87a6a3b05d7adfd2 P 61b887f0d40d497badeca5604c1f0546: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:56:49.302548 15867 tablet_service.cc:1467] Processing CreateTablet for tablet 6be43f8163a043f199d87ec2f969745a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=53f10ccab76d45c3a497fed6e684bef5]), partition=
I20240503 13:56:49.302764 13423 tablet_service.cc:1467] Processing CreateTablet for tablet 6be43f8163a043f199d87ec2f969745a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=53f10ccab76d45c3a497fed6e684bef5]), partition=
I20240503 13:56:49.302801 15867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6be43f8163a043f199d87ec2f969745a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:49.303014 13423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6be43f8163a043f199d87ec2f969745a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:49.303095 22914 tablet_bootstrap.cc:492] T b34b432bb82e465f87a6a3b05d7adfd2 P 61b887f0d40d497badeca5604c1f0546: Bootstrap complete.
I20240503 13:56:49.303267 22914 ts_tablet_manager.cc:1367] T b34b432bb82e465f87a6a3b05d7adfd2 P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.085s user 0.021s sys 0.004s
I20240503 13:56:49.303737 22914 raft_consensus.cc:348] T b34b432bb82e465f87a6a3b05d7adfd2 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:49.303846 22914 raft_consensus.cc:729] T b34b432bb82e465f87a6a3b05d7adfd2 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:49.303974 22914 consensus_queue.cc:260] T b34b432bb82e465f87a6a3b05d7adfd2 P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:49.304872 22914 ts_tablet_manager.cc:1398] T b34b432bb82e465f87a6a3b05d7adfd2 P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:49.305500 13467 tablet_copy_service.cc:342] P 7690ee79577c45e3a2c8f8b552c0bff8: Request end of tablet copy session 61b887f0d40d497badeca5604c1f0546-b34b432bb82e465f87a6a3b05d7adfd2 received from {username='slave'} at 127.6.98.69:40065
I20240503 13:56:49.305552 22935 tablet_bootstrap.cc:492] T 6be43f8163a043f199d87ec2f969745a P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:49.305572 13467 tablet_copy_service.cc:433] P 7690ee79577c45e3a2c8f8b552c0bff8: ending tablet copy session 61b887f0d40d497badeca5604c1f0546-b34b432bb82e465f87a6a3b05d7adfd2 on tablet b34b432bb82e465f87a6a3b05d7adfd2 with peer 61b887f0d40d497badeca5604c1f0546
I20240503 13:56:49.307216 22935 tablet_bootstrap.cc:654] T 6be43f8163a043f199d87ec2f969745a P e47105a0f46e4511b7112e571688a4c7: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:49.308741 22614 consensus_queue.cc:1035] T a7bf405bf11042c394477c63ea98e3b2 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:49.315721 22934 tablet_bootstrap.cc:492] T 6be43f8163a043f199d87ec2f969745a P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:49.316957 22934 tablet_bootstrap.cc:654] T 6be43f8163a043f199d87ec2f969745a P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:49.318374 22933 tablet_bootstrap.cc:492] T 6be43f8163a043f199d87ec2f969745a P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:49.319484 22933 tablet_bootstrap.cc:654] T 6be43f8163a043f199d87ec2f969745a P 61b887f0d40d497badeca5604c1f0546: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:49.322607 22934 tablet_bootstrap.cc:492] T 6be43f8163a043f199d87ec2f969745a P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:49.322785 22934 ts_tablet_manager.cc:1367] T 6be43f8163a043f199d87ec2f969745a P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 13:56:49.323616 22934 raft_consensus.cc:348] T 6be43f8163a043f199d87ec2f969745a P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:49.323832 22934 raft_consensus.cc:374] T 6be43f8163a043f199d87ec2f969745a P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:49.323897 22934 raft_consensus.cc:729] T 6be43f8163a043f199d87ec2f969745a P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:49.324090 22934 consensus_queue.cc:260] T 6be43f8163a043f199d87ec2f969745a P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:49.325062 22933 tablet_bootstrap.cc:492] T 6be43f8163a043f199d87ec2f969745a P 61b887f0d40d497badeca5604c1f0546: No bootstrap required, opened a new log
I20240503 13:56:49.326705 22933 ts_tablet_manager.cc:1367] T 6be43f8163a043f199d87ec2f969745a P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 13:56:49.326840 22934 ts_tablet_manager.cc:1398] T 6be43f8163a043f199d87ec2f969745a P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 13:56:49.327044 22935 tablet_bootstrap.cc:492] T 6be43f8163a043f199d87ec2f969745a P e47105a0f46e4511b7112e571688a4c7: No bootstrap required, opened a new log
I20240503 13:56:49.327143 22935 ts_tablet_manager.cc:1367] T 6be43f8163a043f199d87ec2f969745a P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20240503 13:56:49.327260 22933 raft_consensus.cc:348] T 6be43f8163a043f199d87ec2f969745a P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:49.327369 22933 raft_consensus.cc:374] T 6be43f8163a043f199d87ec2f969745a P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:49.327397 22933 raft_consensus.cc:729] T 6be43f8163a043f199d87ec2f969745a P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:49.327522 22933 consensus_queue.cc:260] T 6be43f8163a043f199d87ec2f969745a P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:49.327880 22935 raft_consensus.cc:348] T 6be43f8163a043f199d87ec2f969745a P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:49.328001 22935 raft_consensus.cc:374] T 6be43f8163a043f199d87ec2f969745a P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:49.328029 22935 raft_consensus.cc:729] T 6be43f8163a043f199d87ec2f969745a P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:49.328140 22935 consensus_queue.cc:260] T 6be43f8163a043f199d87ec2f969745a P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:49.328418 22935 ts_tablet_manager.cc:1398] T 6be43f8163a043f199d87ec2f969745a P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:49.330500 22933 ts_tablet_manager.cc:1398] T 6be43f8163a043f199d87ec2f969745a P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 13:56:49.341171 22614 consensus_queue.cc:1035] T a7bf405bf11042c394477c63ea98e3b2 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:49.410708 21492 raft_consensus.cc:482] T 6be43f8163a043f199d87ec2f969745a P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:49.410835 21492 raft_consensus.cc:504] T 6be43f8163a043f199d87ec2f969745a P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:49.411196 21492 leader_election.cc:288] T 6be43f8163a043f199d87ec2f969745a P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:49.411701 15889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6be43f8163a043f199d87ec2f969745a" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7" is_pre_election: true
I20240503 13:56:49.411837 15889 raft_consensus.cc:2454] T 6be43f8163a043f199d87ec2f969745a P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 0.
I20240503 13:56:49.411841 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6be43f8163a043f199d87ec2f969745a" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
I20240503 13:56:49.411969 13445 raft_consensus.cc:2454] T 6be43f8163a043f199d87ec2f969745a P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 0.
I20240503 13:56:49.412757 15426 leader_election.cc:302] T 6be43f8163a043f199d87ec2f969745a P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61b887f0d40d497badeca5604c1f0546, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:49.413286 20722 raft_consensus.cc:2781] T 6be43f8163a043f199d87ec2f969745a P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:49.413344 20722 raft_consensus.cc:482] T 6be43f8163a043f199d87ec2f969745a P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:49.413399 20722 raft_consensus.cc:3037] T 6be43f8163a043f199d87ec2f969745a P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:49.414557 20722 raft_consensus.cc:504] T 6be43f8163a043f199d87ec2f969745a P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:49.414887 20722 leader_election.cc:288] T 6be43f8163a043f199d87ec2f969745a P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 election: Requested vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:49.416039 15888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6be43f8163a043f199d87ec2f969745a" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7"
I20240503 13:56:49.416168 15888 raft_consensus.cc:3037] T 6be43f8163a043f199d87ec2f969745a P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:49.416682 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6be43f8163a043f199d87ec2f969745a" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
I20240503 13:56:49.416798 13445 raft_consensus.cc:3037] T 6be43f8163a043f199d87ec2f969745a P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:49.417233 15888 raft_consensus.cc:2454] T 6be43f8163a043f199d87ec2f969745a P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 1.
I20240503 13:56:49.417531 13445 raft_consensus.cc:2454] T 6be43f8163a043f199d87ec2f969745a P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 1.
I20240503 13:56:49.417577 15426 leader_election.cc:302] T 6be43f8163a043f199d87ec2f969745a P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61b887f0d40d497badeca5604c1f0546, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:49.417768 21492 raft_consensus.cc:2781] T 6be43f8163a043f199d87ec2f969745a P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:49.417843 21492 raft_consensus.cc:686] T 6be43f8163a043f199d87ec2f969745a P 61b887f0d40d497badeca5604c1f0546 [term 1 LEADER]: Becoming Leader. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Running, Role: LEADER
I20240503 13:56:49.417969 21492 consensus_queue.cc:237] T 6be43f8163a043f199d87ec2f969745a P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:49.420207 12912 catalog_manager.cc:5484] T 6be43f8163a043f199d87ec2f969745a P 61b887f0d40d497badeca5604c1f0546 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61b887f0d40d497badeca5604c1f0546 (127.6.98.69). New cstate: current_term: 1 leader_uuid: "61b887f0d40d497badeca5604c1f0546" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } }
I20240503 13:56:49.429239 12910 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_extra_table_52"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:56:49.436790 13032 tablet_service.cc:1467] Processing CreateTablet for tablet 013d4904c1ce48b78884dd61d9dacb32 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=f0a7b6a78ce043c3bf68403274e60d4d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:49.437129 13032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 013d4904c1ce48b78884dd61d9dacb32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:49.437346 15738 tablet_service.cc:1467] Processing CreateTablet for tablet 17ef7be98ce84be2a0a8ac921a0e479d (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=f0a7b6a78ce043c3bf68403274e60d4d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:49.437415 13163 tablet_service.cc:1467] Processing CreateTablet for tablet 17ef7be98ce84be2a0a8ac921a0e479d (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=f0a7b6a78ce043c3bf68403274e60d4d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:49.437641 15738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17ef7be98ce84be2a0a8ac921a0e479d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:49.437671 13163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17ef7be98ce84be2a0a8ac921a0e479d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:49.439677 22986 tablet_bootstrap.cc:492] T 013d4904c1ce48b78884dd61d9dacb32 P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:49.439723 13164 tablet_service.cc:1467] Processing CreateTablet for tablet 013d4904c1ce48b78884dd61d9dacb32 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=f0a7b6a78ce043c3bf68403274e60d4d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:49.439944 13164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 013d4904c1ce48b78884dd61d9dacb32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:49.440065 15472 tablet_service.cc:1467] Processing CreateTablet for tablet 24b7efa8295b42668dc31a8083272924 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=f0a7b6a78ce043c3bf68403274e60d4d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:49.440330 15472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24b7efa8295b42668dc31a8083272924. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:49.441147 22986 tablet_bootstrap.cc:654] T 013d4904c1ce48b78884dd61d9dacb32 P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:49.442086 15736 tablet_service.cc:1467] Processing CreateTablet for tablet 24b7efa8295b42668dc31a8083272924 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=f0a7b6a78ce043c3bf68403274e60d4d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:49.442292 22986 tablet_bootstrap.cc:492] T 013d4904c1ce48b78884dd61d9dacb32 P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:49.442314 15736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24b7efa8295b42668dc31a8083272924. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:49.442371 22986 ts_tablet_manager.cc:1367] T 013d4904c1ce48b78884dd61d9dacb32 P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:56:49.442417 22933 tablet_bootstrap.cc:492] T 24b7efa8295b42668dc31a8083272924 P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:49.442801 13296 tablet_service.cc:1467] Processing CreateTablet for tablet 013d4904c1ce48b78884dd61d9dacb32 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=f0a7b6a78ce043c3bf68403274e60d4d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:49.442826 22986 raft_consensus.cc:348] T 013d4904c1ce48b78884dd61d9dacb32 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:49.442931 22986 raft_consensus.cc:374] T 013d4904c1ce48b78884dd61d9dacb32 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:49.442957 22986 raft_consensus.cc:729] T 013d4904c1ce48b78884dd61d9dacb32 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:49.443091 13296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 013d4904c1ce48b78884dd61d9dacb32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:49.443094 22986 consensus_queue.cc:260] T 013d4904c1ce48b78884dd61d9dacb32 P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:49.443372 22986 ts_tablet_manager.cc:1398] T 013d4904c1ce48b78884dd61d9dacb32 P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:49.443413 22933 tablet_bootstrap.cc:654] T 24b7efa8295b42668dc31a8083272924 P 61b887f0d40d497badeca5604c1f0546: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:49.445634 13034 tablet_service.cc:1467] Processing CreateTablet for tablet 24b7efa8295b42668dc31a8083272924 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=f0a7b6a78ce043c3bf68403274e60d4d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:49.445828 13032 tablet_service.cc:1467] Processing CreateTablet for tablet 17ef7be98ce84be2a0a8ac921a0e479d (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=f0a7b6a78ce043c3bf68403274e60d4d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:49.445904 13034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24b7efa8295b42668dc31a8083272924. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:49.446060 13032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17ef7be98ce84be2a0a8ac921a0e479d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:49.447062 22933 tablet_bootstrap.cc:492] T 24b7efa8295b42668dc31a8083272924 P 61b887f0d40d497badeca5604c1f0546: No bootstrap required, opened a new log
I20240503 13:56:49.447153 22933 ts_tablet_manager.cc:1367] T 24b7efa8295b42668dc31a8083272924 P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:56:49.447602 22933 raft_consensus.cc:348] T 24b7efa8295b42668dc31a8083272924 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:49.447736 22933 raft_consensus.cc:374] T 24b7efa8295b42668dc31a8083272924 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:49.447782 22933 raft_consensus.cc:729] T 24b7efa8295b42668dc31a8083272924 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:49.447918 22933 consensus_queue.cc:260] T 24b7efa8295b42668dc31a8083272924 P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:49.448241 22933 ts_tablet_manager.cc:1398] T 24b7efa8295b42668dc31a8083272924 P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:49.448596 22986 tablet_bootstrap.cc:492] T 24b7efa8295b42668dc31a8083272924 P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:49.449527 22986 tablet_bootstrap.cc:654] T 24b7efa8295b42668dc31a8083272924 P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:49.451409 22991 tablet_bootstrap.cc:492] T 013d4904c1ce48b78884dd61d9dacb32 P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:49.452316 22991 tablet_bootstrap.cc:654] T 013d4904c1ce48b78884dd61d9dacb32 P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:49.453109 22987 tablet_bootstrap.cc:492] T 17ef7be98ce84be2a0a8ac921a0e479d P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:49.453445 22988 tablet_bootstrap.cc:492] T 17ef7be98ce84be2a0a8ac921a0e479d P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:49.454442 22991 tablet_bootstrap.cc:492] T 013d4904c1ce48b78884dd61d9dacb32 P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:49.454530 22991 ts_tablet_manager.cc:1367] T 013d4904c1ce48b78884dd61d9dacb32 P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:56:49.455034 22991 raft_consensus.cc:348] T 013d4904c1ce48b78884dd61d9dacb32 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:49.455143 22991 raft_consensus.cc:374] T 013d4904c1ce48b78884dd61d9dacb32 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:49.455175 22991 raft_consensus.cc:729] T 013d4904c1ce48b78884dd61d9dacb32 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:49.455304 22991 consensus_queue.cc:260] T 013d4904c1ce48b78884dd61d9dacb32 P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:49.455592 22991 ts_tablet_manager.cc:1398] T 013d4904c1ce48b78884dd61d9dacb32 P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:49.455639 22986 tablet_bootstrap.cc:492] T 24b7efa8295b42668dc31a8083272924 P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:49.455734 22986 ts_tablet_manager.cc:1367] T 24b7efa8295b42668dc31a8083272924 P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240503 13:56:49.455816 22987 tablet_bootstrap.cc:654] T 17ef7be98ce84be2a0a8ac921a0e479d P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:49.456207 22986 raft_consensus.cc:348] T 24b7efa8295b42668dc31a8083272924 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:49.456308 22986 raft_consensus.cc:374] T 24b7efa8295b42668dc31a8083272924 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:49.456341 22986 raft_consensus.cc:729] T 24b7efa8295b42668dc31a8083272924 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:49.456478 22986 consensus_queue.cc:260] T 24b7efa8295b42668dc31a8083272924 P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:49.458206 22738 raft_consensus.cc:482] T 24b7efa8295b42668dc31a8083272924 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:49.458292 22738 raft_consensus.cc:504] T 24b7efa8295b42668dc31a8083272924 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:49.458365 22986 ts_tablet_manager.cc:1398] T 24b7efa8295b42668dc31a8083272924 P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:56:49.458521 22986 tablet_bootstrap.cc:492] T 17ef7be98ce84be2a0a8ac921a0e479d P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:49.458734 22738 leader_election.cc:288] T 24b7efa8295b42668dc31a8083272924 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885), 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153)
I20240503 13:56:49.459131 15758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24b7efa8295b42668dc31a8083272924" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2" is_pre_election: true
I20240503 13:56:49.459448 22986 tablet_bootstrap.cc:654] T 17ef7be98ce84be2a0a8ac921a0e479d P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:49.459653 15492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24b7efa8295b42668dc31a8083272924" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546" is_pre_election: true
I20240503 13:56:49.459777 15492 raft_consensus.cc:2454] T 24b7efa8295b42668dc31a8083272924 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 0.
W20240503 13:56:49.459978 12990 leader_election.cc:341] T 24b7efa8295b42668dc31a8083272924 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885): Illegal state: must be running to vote when last-logged opid is not known
I20240503 13:56:49.460207 12990 leader_election.cc:302] T 24b7efa8295b42668dc31a8083272924 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 61b887f0d40d497badeca5604c1f0546; no voters: 76c386bef61a4bd5a907266a5118ffe2
I20240503 13:56:49.460425 22738 raft_consensus.cc:2781] T 24b7efa8295b42668dc31a8083272924 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:49.460517 22738 raft_consensus.cc:482] T 24b7efa8295b42668dc31a8083272924 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:49.460558 22738 raft_consensus.cc:3037] T 24b7efa8295b42668dc31a8083272924 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:49.461670 22738 raft_consensus.cc:504] T 24b7efa8295b42668dc31a8083272924 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:49.462033 22738 leader_election.cc:288] T 24b7efa8295b42668dc31a8083272924 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 election: Requested vote from peers 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885), 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153)
I20240503 13:56:49.462692 22996 ts_tablet_manager.cc:902] T c7d5e8c09ec14ac7891c8a63cb896878 P e47105a0f46e4511b7112e571688a4c7: Initiating tablet copy from peer 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153)
I20240503 13:56:49.463203 15758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24b7efa8295b42668dc31a8083272924" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2"
I20240503 13:56:49.463239 22996 tablet_copy_client.cc:308] T c7d5e8c09ec14ac7891c8a63cb896878 P e47105a0f46e4511b7112e571688a4c7: tablet copy: Beginning tablet copy session from remote peer at address 127.6.98.69:39153
W20240503 13:56:49.463721 12990 leader_election.cc:341] T 24b7efa8295b42668dc31a8083272924 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885): Illegal state: must be running to vote when last-logged opid is not known
I20240503 13:56:49.463719 15513 tablet_copy_service.cc:140] P 61b887f0d40d497badeca5604c1f0546: Received BeginTabletCopySession request for tablet c7d5e8c09ec14ac7891c8a63cb896878 from peer e47105a0f46e4511b7112e571688a4c7 ({username='slave'} at 127.6.98.72:53521)
I20240503 13:56:49.463822 15513 tablet_copy_service.cc:161] P 61b887f0d40d497badeca5604c1f0546: Beginning new tablet copy session on tablet c7d5e8c09ec14ac7891c8a63cb896878 from peer e47105a0f46e4511b7112e571688a4c7 at {username='slave'} at 127.6.98.72:53521: session id = e47105a0f46e4511b7112e571688a4c7-c7d5e8c09ec14ac7891c8a63cb896878
I20240503 13:56:49.463909 15492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24b7efa8295b42668dc31a8083272924" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546"
I20240503 13:56:49.464028 15492 raft_consensus.cc:3037] T 24b7efa8295b42668dc31a8083272924 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:49.464689 15513 tablet_copy_source_session.cc:205] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:56:49.465140 15492 raft_consensus.cc:2454] T 24b7efa8295b42668dc31a8083272924 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 1.
I20240503 13:56:49.466068 12990 leader_election.cc:302] T 24b7efa8295b42668dc31a8083272924 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 61b887f0d40d497badeca5604c1f0546; no voters: 76c386bef61a4bd5a907266a5118ffe2
I20240503 13:56:49.468319 22738 raft_consensus.cc:2781] T 24b7efa8295b42668dc31a8083272924 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:49.468410 22738 raft_consensus.cc:686] T 24b7efa8295b42668dc31a8083272924 P 60ba3ffe3b7746469834cd91c84be14e [term 1 LEADER]: Becoming Leader. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Running, Role: LEADER
I20240503 13:56:49.468570 22738 consensus_queue.cc:237] T 24b7efa8295b42668dc31a8083272924 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:49.470067 22988 tablet_bootstrap.cc:654] T 17ef7be98ce84be2a0a8ac921a0e479d P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:49.471804 22996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7d5e8c09ec14ac7891c8a63cb896878. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:49.471787 12908 catalog_manager.cc:5484] T 24b7efa8295b42668dc31a8083272924 P 60ba3ffe3b7746469834cd91c84be14e reported cstate change: term changed from 0 to 1, leader changed from <none> to 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65). New cstate: current_term: 1 leader_uuid: "60ba3ffe3b7746469834cd91c84be14e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } }
I20240503 13:56:49.475243 22986 tablet_bootstrap.cc:492] T 17ef7be98ce84be2a0a8ac921a0e479d P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:49.475343 22986 ts_tablet_manager.cc:1367] T 17ef7be98ce84be2a0a8ac921a0e479d P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.001s
I20240503 13:56:49.475399 22987 tablet_bootstrap.cc:492] T 17ef7be98ce84be2a0a8ac921a0e479d P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:49.475495 22987 ts_tablet_manager.cc:1367] T 17ef7be98ce84be2a0a8ac921a0e479d P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20240503 13:56:49.475816 22986 raft_consensus.cc:348] T 17ef7be98ce84be2a0a8ac921a0e479d P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:49.475939 22986 raft_consensus.cc:374] T 17ef7be98ce84be2a0a8ac921a0e479d P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:49.475975 22986 raft_consensus.cc:729] T 17ef7be98ce84be2a0a8ac921a0e479d P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:49.476083 22986 consensus_queue.cc:260] T 17ef7be98ce84be2a0a8ac921a0e479d P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:49.476212 22987 raft_consensus.cc:348] T 17ef7be98ce84be2a0a8ac921a0e479d P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:49.476316 22987 raft_consensus.cc:374] T 17ef7be98ce84be2a0a8ac921a0e479d P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:49.476347 22987 raft_consensus.cc:729] T 17ef7be98ce84be2a0a8ac921a0e479d P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:49.476394 22986 ts_tablet_manager.cc:1398] T 17ef7be98ce84be2a0a8ac921a0e479d P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:49.476511 22987 consensus_queue.cc:260] T 17ef7be98ce84be2a0a8ac921a0e479d P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:49.478646 22996 tablet_copy_client.cc:783] T c7d5e8c09ec14ac7891c8a63cb896878 P e47105a0f46e4511b7112e571688a4c7: tablet copy: Starting download of 0 data blocks...
I20240503 13:56:49.478863 22996 tablet_copy_client.cc:647] T c7d5e8c09ec14ac7891c8a63cb896878 P e47105a0f46e4511b7112e571688a4c7: tablet copy: Starting download of 1 WAL segments...
I20240503 13:56:49.477754 22987 ts_tablet_manager.cc:1398] T 17ef7be98ce84be2a0a8ac921a0e479d P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:49.479796 22987 tablet_bootstrap.cc:492] T 013d4904c1ce48b78884dd61d9dacb32 P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:49.480681 22996 tablet_copy_client.cc:523] T c7d5e8c09ec14ac7891c8a63cb896878 P e47105a0f46e4511b7112e571688a4c7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:56:49.481025 22987 tablet_bootstrap.cc:654] T 013d4904c1ce48b78884dd61d9dacb32 P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:49.482226 22996 tablet_bootstrap.cc:492] T c7d5e8c09ec14ac7891c8a63cb896878 P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:49.485436 22987 tablet_bootstrap.cc:492] T 013d4904c1ce48b78884dd61d9dacb32 P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:49.485529 22987 ts_tablet_manager.cc:1367] T 013d4904c1ce48b78884dd61d9dacb32 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 13:56:49.485992 22987 raft_consensus.cc:348] T 013d4904c1ce48b78884dd61d9dacb32 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:49.486110 22987 raft_consensus.cc:374] T 013d4904c1ce48b78884dd61d9dacb32 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:49.486152 22987 raft_consensus.cc:729] T 013d4904c1ce48b78884dd61d9dacb32 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:49.486316 22996 tablet_bootstrap.cc:492] T c7d5e8c09ec14ac7891c8a63cb896878 P e47105a0f46e4511b7112e571688a4c7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:56:49.486277 22987 consensus_queue.cc:260] T 013d4904c1ce48b78884dd61d9dacb32 P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:49.486692 22987 ts_tablet_manager.cc:1398] T 013d4904c1ce48b78884dd61d9dacb32 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:49.487244 22996 tablet_bootstrap.cc:492] T c7d5e8c09ec14ac7891c8a63cb896878 P e47105a0f46e4511b7112e571688a4c7: Bootstrap complete.
I20240503 13:56:49.487406 22996 ts_tablet_manager.cc:1367] T c7d5e8c09ec14ac7891c8a63cb896878 P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 13:56:49.487934 22996 raft_consensus.cc:348] T c7d5e8c09ec14ac7891c8a63cb896878 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:49.488052 22996 raft_consensus.cc:729] T c7d5e8c09ec14ac7891c8a63cb896878 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:49.488188 22996 consensus_queue.cc:260] T c7d5e8c09ec14ac7891c8a63cb896878 P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:49.488719 22996 ts_tablet_manager.cc:1398] T c7d5e8c09ec14ac7891c8a63cb896878 P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:49.489511 15513 tablet_copy_service.cc:342] P 61b887f0d40d497badeca5604c1f0546: Request end of tablet copy session e47105a0f46e4511b7112e571688a4c7-c7d5e8c09ec14ac7891c8a63cb896878 received from {username='slave'} at 127.6.98.72:53521
I20240503 13:56:49.489593 15513 tablet_copy_service.cc:433] P 61b887f0d40d497badeca5604c1f0546: ending tablet copy session e47105a0f46e4511b7112e571688a4c7-c7d5e8c09ec14ac7891c8a63cb896878 on tablet c7d5e8c09ec14ac7891c8a63cb896878 with peer e47105a0f46e4511b7112e571688a4c7
I20240503 13:56:49.491678 22988 tablet_bootstrap.cc:492] T 17ef7be98ce84be2a0a8ac921a0e479d P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:49.491782 22988 ts_tablet_manager.cc:1367] T 17ef7be98ce84be2a0a8ac921a0e479d P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.038s user 0.002s sys 0.000s
I20240503 13:56:49.493310 22988 raft_consensus.cc:348] T 17ef7be98ce84be2a0a8ac921a0e479d P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:49.493425 22988 raft_consensus.cc:374] T 17ef7be98ce84be2a0a8ac921a0e479d P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:49.493456 22988 raft_consensus.cc:729] T 17ef7be98ce84be2a0a8ac921a0e479d P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:49.493571 22988 consensus_queue.cc:260] T 17ef7be98ce84be2a0a8ac921a0e479d P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:49.493921 22988 ts_tablet_manager.cc:1398] T 17ef7be98ce84be2a0a8ac921a0e479d P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:49.494118 22988 tablet_bootstrap.cc:492] T 24b7efa8295b42668dc31a8083272924 P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:49.496126 22988 tablet_bootstrap.cc:654] T 24b7efa8295b42668dc31a8083272924 P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:49.497813 22996 ts_tablet_manager.cc:902] T 604875eff3594876af54e9126057bd7f P e47105a0f46e4511b7112e571688a4c7: Initiating tablet copy from peer 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885)
I20240503 13:56:49.498194 22988 tablet_bootstrap.cc:492] T 24b7efa8295b42668dc31a8083272924 P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:49.498240 22996 tablet_copy_client.cc:308] T 604875eff3594876af54e9126057bd7f P e47105a0f46e4511b7112e571688a4c7: tablet copy: Beginning tablet copy session from remote peer at address 127.6.98.71:42885
I20240503 13:56:49.498298 22988 ts_tablet_manager.cc:1367] T 24b7efa8295b42668dc31a8083272924 P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:56:49.498648 15778 tablet_copy_service.cc:140] P 76c386bef61a4bd5a907266a5118ffe2: Received BeginTabletCopySession request for tablet 604875eff3594876af54e9126057bd7f from peer e47105a0f46e4511b7112e571688a4c7 ({username='slave'} at 127.6.98.72:57601)
I20240503 13:56:49.498736 15778 tablet_copy_service.cc:161] P 76c386bef61a4bd5a907266a5118ffe2: Beginning new tablet copy session on tablet 604875eff3594876af54e9126057bd7f from peer e47105a0f46e4511b7112e571688a4c7 at {username='slave'} at 127.6.98.72:57601: session id = e47105a0f46e4511b7112e571688a4c7-604875eff3594876af54e9126057bd7f
I20240503 13:56:49.498814 22988 raft_consensus.cc:348] T 24b7efa8295b42668dc31a8083272924 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:49.498934 22988 raft_consensus.cc:374] T 24b7efa8295b42668dc31a8083272924 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:49.498967 22988 raft_consensus.cc:729] T 24b7efa8295b42668dc31a8083272924 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:49.499110 22988 consensus_queue.cc:260] T 24b7efa8295b42668dc31a8083272924 P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:49.499635 15778 tablet_copy_source_session.cc:205] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:56:49.500995 22996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 604875eff3594876af54e9126057bd7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:49.501293 22988 ts_tablet_manager.cc:1398] T 24b7efa8295b42668dc31a8083272924 P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:56:49.504133 22996 tablet_copy_client.cc:783] T 604875eff3594876af54e9126057bd7f P e47105a0f46e4511b7112e571688a4c7: tablet copy: Starting download of 0 data blocks...
I20240503 13:56:49.504294 22996 tablet_copy_client.cc:647] T 604875eff3594876af54e9126057bd7f P e47105a0f46e4511b7112e571688a4c7: tablet copy: Starting download of 1 WAL segments...
I20240503 13:56:49.509332 15605 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:56:49.514809 22996 tablet_copy_client.cc:523] T 604875eff3594876af54e9126057bd7f P e47105a0f46e4511b7112e571688a4c7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:56:49.516389 22996 tablet_bootstrap.cc:492] T 604875eff3594876af54e9126057bd7f P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:49.518608 13423 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:49.520017 22996 tablet_bootstrap.cc:492] T 604875eff3594876af54e9126057bd7f P e47105a0f46e4511b7112e571688a4c7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:56:49.520251 22996 tablet_bootstrap.cc:492] T 604875eff3594876af54e9126057bd7f P e47105a0f46e4511b7112e571688a4c7: Bootstrap complete.
I20240503 13:56:49.520411 22996 ts_tablet_manager.cc:1367] T 604875eff3594876af54e9126057bd7f P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240503 13:56:49.520941 22996 raft_consensus.cc:348] T 604875eff3594876af54e9126057bd7f P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:49.521076 22996 raft_consensus.cc:729] T 604875eff3594876af54e9126057bd7f P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:49.521227 22996 consensus_queue.cc:260] T 604875eff3594876af54e9126057bd7f P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:49.521617 22996 ts_tablet_manager.cc:1398] T 604875eff3594876af54e9126057bd7f P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 13:56:49.523319 15472 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 13:56:49.528064 15778 tablet_copy_service.cc:342] P 76c386bef61a4bd5a907266a5118ffe2: Request end of tablet copy session e47105a0f46e4511b7112e571688a4c7-604875eff3594876af54e9126057bd7f received from {username='slave'} at 127.6.98.72:57601
I20240503 13:56:49.528160 15778 tablet_copy_service.cc:433] P 76c386bef61a4bd5a907266a5118ffe2: ending tablet copy session e47105a0f46e4511b7112e571688a4c7-604875eff3594876af54e9126057bd7f on tablet 604875eff3594876af54e9126057bd7f with peer e47105a0f46e4511b7112e571688a4c7
I20240503 13:56:49.531239 13032 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:56:49.531522 15736 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:56:49.535202 12906 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:56:49.538810 20870 raft_consensus.cc:482] T 013d4904c1ce48b78884dd61d9dacb32 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:49.538933 20870 raft_consensus.cc:504] T 013d4904c1ce48b78884dd61d9dacb32 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:49.539391 20870 leader_election.cc:288] T 013d4904c1ce48b78884dd61d9dacb32 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:49.539554 22421 raft_consensus.cc:482] T 17ef7be98ce84be2a0a8ac921a0e479d P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:49.539650 22421 raft_consensus.cc:504] T 17ef7be98ce84be2a0a8ac921a0e479d P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:49.540097 22421 leader_election.cc:288] T 17ef7be98ce84be2a0a8ac921a0e479d P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695), 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885)
I20240503 13:56:49.540342 13183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "013d4904c1ce48b78884dd61d9dacb32" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770" is_pre_election: true
I20240503 13:56:49.540427 13296 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 13:56:49.540447 13183 raft_consensus.cc:2454] T 013d4904c1ce48b78884dd61d9dacb32 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 0.
I20240503 13:56:49.540776 13314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "013d4904c1ce48b78884dd61d9dacb32" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703" is_pre_election: true
I20240503 13:56:49.540897 13314 raft_consensus.cc:2454] T 013d4904c1ce48b78884dd61d9dacb32 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 0.
I20240503 13:56:49.541383 12990 leader_election.cc:302] T 013d4904c1ce48b78884dd61d9dacb32 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:49.542296 13445 raft_consensus.cc:1261] T 6be43f8163a043f199d87ec2f969745a P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer 61b887f0d40d497badeca5604c1f0546: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:49.542412 13164 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 13:56:49.542634 15758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17ef7be98ce84be2a0a8ac921a0e479d" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2" is_pre_election: true
I20240503 13:56:49.542685 20722 consensus_queue.cc:1035] T 6be43f8163a043f199d87ec2f969745a P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:49.542774 15758 raft_consensus.cc:2454] T 17ef7be98ce84be2a0a8ac921a0e479d P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 0.
I20240503 13:56:49.543252 13121 leader_election.cc:302] T 17ef7be98ce84be2a0a8ac921a0e479d P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76c386bef61a4bd5a907266a5118ffe2, 8ccf5a4ac450414c9d3579bd5800b770; no voters:
I20240503 13:56:49.543514 22421 raft_consensus.cc:2781] T 17ef7be98ce84be2a0a8ac921a0e479d P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:49.543560 22421 raft_consensus.cc:482] T 17ef7be98ce84be2a0a8ac921a0e479d P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:49.543592 22421 raft_consensus.cc:3037] T 17ef7be98ce84be2a0a8ac921a0e479d P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:49.544814 22421 raft_consensus.cc:504] T 17ef7be98ce84be2a0a8ac921a0e479d P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:49.544880 20870 raft_consensus.cc:2781] T 013d4904c1ce48b78884dd61d9dacb32 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:49.544927 20870 raft_consensus.cc:482] T 013d4904c1ce48b78884dd61d9dacb32 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:49.544961 20870 raft_consensus.cc:3037] T 013d4904c1ce48b78884dd61d9dacb32 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:49.545166 22421 leader_election.cc:288] T 17ef7be98ce84be2a0a8ac921a0e479d P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 election: Requested vote from peers 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695), 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885)
I20240503 13:56:49.545488 12906 catalog_manager.cc:5576] TS 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109) does not have the latest schema for tablet 6be43f8163a043f199d87ec2f969745a (table rebalancer_dynamic_table [id=53f10ccab76d45c3a497fed6e684bef5]). Expected version 1 got 0
I20240503 13:56:49.545871 13052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17ef7be98ce84be2a0a8ac921a0e479d" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e"
I20240503 13:56:49.545979 13052 raft_consensus.cc:3037] T 17ef7be98ce84be2a0a8ac921a0e479d P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:49.545981 20870 raft_consensus.cc:504] T 013d4904c1ce48b78884dd61d9dacb32 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:49.546319 20870 leader_election.cc:288] T 013d4904c1ce48b78884dd61d9dacb32 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 election: Requested vote from peers ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:49.546629 13314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "013d4904c1ce48b78884dd61d9dacb32" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703"
I20240503 13:56:49.546731 13314 raft_consensus.cc:3037] T 013d4904c1ce48b78884dd61d9dacb32 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:49.546896 13052 raft_consensus.cc:2454] T 17ef7be98ce84be2a0a8ac921a0e479d P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 1.
I20240503 13:56:49.547000 15888 raft_consensus.cc:1261] T 6be43f8163a043f199d87ec2f969745a P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Refusing update from remote peer 61b887f0d40d497badeca5604c1f0546: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:49.547655 13314 raft_consensus.cc:2454] T 013d4904c1ce48b78884dd61d9dacb32 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 1.
I20240503 13:56:49.547927 12990 leader_election.cc:302] T 013d4904c1ce48b78884dd61d9dacb32 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:49.548087 20870 raft_consensus.cc:2781] T 013d4904c1ce48b78884dd61d9dacb32 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:49.548058 20722 consensus_queue.cc:1035] T 6be43f8163a043f199d87ec2f969745a P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:49.548154 20870 raft_consensus.cc:686] T 013d4904c1ce48b78884dd61d9dacb32 P 60ba3ffe3b7746469834cd91c84be14e [term 1 LEADER]: Becoming Leader. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Running, Role: LEADER
I20240503 13:56:49.548270 20870 consensus_queue.cc:237] T 013d4904c1ce48b78884dd61d9dacb32 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:49.550554 13051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17ef7be98ce84be2a0a8ac921a0e479d" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e" is_pre_election: true
I20240503 13:56:49.550663 13051 raft_consensus.cc:2362] T 17ef7be98ce84be2a0a8ac921a0e479d P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 1. Re-sending same reply.
I20240503 13:56:49.552196 12906 catalog_manager.cc:5576] TS e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753) does not have the latest schema for tablet 6be43f8163a043f199d87ec2f969745a (table rebalancer_dynamic_table [id=53f10ccab76d45c3a497fed6e684bef5]). Expected version 1 got 0
I20240503 13:56:49.553560 13183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "013d4904c1ce48b78884dd61d9dacb32" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770"
I20240503 13:56:49.553680 13183 raft_consensus.cc:3037] T 013d4904c1ce48b78884dd61d9dacb32 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:49.554654 13183 raft_consensus.cc:2454] T 013d4904c1ce48b78884dd61d9dacb32 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 1.
I20240503 13:56:49.555159 13119 leader_election.cc:302] T 17ef7be98ce84be2a0a8ac921a0e479d P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 8ccf5a4ac450414c9d3579bd5800b770; no voters:
I20240503 13:56:49.555766 12906 catalog_manager.cc:5484] T 013d4904c1ce48b78884dd61d9dacb32 P 60ba3ffe3b7746469834cd91c84be14e reported cstate change: term changed from 0 to 1, leader changed from <none> to 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65). New cstate: current_term: 1 leader_uuid: "60ba3ffe3b7746469834cd91c84be14e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } }
I20240503 13:56:49.556668 23025 raft_consensus.cc:2781] T 17ef7be98ce84be2a0a8ac921a0e479d P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:49.556972 15758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17ef7be98ce84be2a0a8ac921a0e479d" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2"
I20240503 13:56:49.557053 23025 raft_consensus.cc:686] T 17ef7be98ce84be2a0a8ac921a0e479d P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 LEADER]: Becoming Leader. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Running, Role: LEADER
I20240503 13:56:49.557108 15758 raft_consensus.cc:3037] T 17ef7be98ce84be2a0a8ac921a0e479d P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:49.557206 23025 consensus_queue.cc:237] T 17ef7be98ce84be2a0a8ac921a0e479d P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:49.558168 15758 raft_consensus.cc:2454] T 17ef7be98ce84be2a0a8ac921a0e479d P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 1.
I20240503 13:56:49.560259 12908 catalog_manager.cc:5484] T 17ef7be98ce84be2a0a8ac921a0e479d P 8ccf5a4ac450414c9d3579bd5800b770 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66). New cstate: current_term: 1 leader_uuid: "8ccf5a4ac450414c9d3579bd5800b770" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } }
I20240503 13:56:49.561543 15867 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:56:49.588742 23023 mvcc.cc:205] Tried to move back new op lower bound from 7023593920680075264 to 7023593920177623040. Current Snapshot: MvccSnapshot[applied={T|T < 7023593920680075264}]
I20240503 13:56:49.592793 12908 catalog_manager.cc:5576] TS e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753) does not have the latest schema for tablet 6be43f8163a043f199d87ec2f969745a (table rebalancer_dynamic_table [id=53f10ccab76d45c3a497fed6e684bef5]). Expected version 1 got 0
I20240503 13:56:49.603731 23027 mvcc.cc:205] Tried to move back new op lower bound from 7023593920680075264 to 7023593920177623040. Current Snapshot: MvccSnapshot[applied={T|T < 7023593920680075264}]
I20240503 13:56:49.603968 23032 tablet.cc:1722] T 6be43f8163a043f199d87ec2f969745a P e47105a0f46e4511b7112e571688a4c7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:49.604068 23031 tablet.cc:1722] T 6be43f8163a043f199d87ec2f969745a P 61b887f0d40d497badeca5604c1f0546: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:49.604396 23031 tablet.cc:1620] T 6be43f8163a043f199d87ec2f969745a P 61b887f0d40d497badeca5604c1f0546: MemRowSet was empty: no flush needed.
I20240503 13:56:49.604429 23032 tablet.cc:1620] T 6be43f8163a043f199d87ec2f969745a P e47105a0f46e4511b7112e571688a4c7: MemRowSet was empty: no flush needed.
I20240503 13:56:49.610193 12908 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=53f10ccab76d45c3a497fed6e684bef5] alter complete (version 1)
I20240503 13:56:49.610724 12906 catalog_manager.cc:5576] TS 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109) does not have the latest schema for tablet 6be43f8163a043f199d87ec2f969745a (table rebalancer_dynamic_table [id=53f10ccab76d45c3a497fed6e684bef5]). Expected version 1 got 0
I20240503 13:56:49.623898 23033 tablet.cc:1722] T 6be43f8163a043f199d87ec2f969745a P 7690ee79577c45e3a2c8f8b552c0bff8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:49.625025 23033 tablet.cc:1620] T 6be43f8163a043f199d87ec2f969745a P 7690ee79577c45e3a2c8f8b552c0bff8: MemRowSet was empty: no flush needed.
I20240503 13:56:49.650040 15888 consensus_queue.cc:237] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:49.651396 15624 raft_consensus.cc:1261] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Refusing update from remote peer e47105a0f46e4511b7112e571688a4c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:56:49.651461 13314 raft_consensus.cc:1261] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Refusing update from remote peer e47105a0f46e4511b7112e571688a4c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:56:49.651914 22630 consensus_queue.cc:1035] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:56:49.652127 22256 consensus_queue.cc:1035] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:56:49.654436 22256 raft_consensus.cc:2932] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P e47105a0f46e4511b7112e571688a4c7 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:49.654814 15624 raft_consensus.cc:2932] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:49.656096 13314 raft_consensus.cc:2932] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:49.657850 12906 catalog_manager.cc:5484] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P e47105a0f46e4511b7112e571688a4c7 reported cstate change: config changed from index 2 to 3, VOTER 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71) evicted. New cstate: current_term: 1 leader_uuid: "e47105a0f46e4511b7112e571688a4c7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:49.660876 15736 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bdb9c14caf04abca39e56f8bcf1d1d4 with delete_type TABLET_DATA_TOMBSTONED (TS 76c386bef61a4bd5a907266a5118ffe2 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:44984
I20240503 13:56:49.661142 22913 tablet_replica.cc:330] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 76c386bef61a4bd5a907266a5118ffe2: stopping tablet replica
I20240503 13:56:49.661307 22913 raft_consensus.cc:2229] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:49.661439 22913 raft_consensus.cc:2258] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:49.661862 22913 ts_tablet_manager.cc:1875] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 76c386bef61a4bd5a907266a5118ffe2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:56:49.663918 22913 ts_tablet_manager.cc:1888] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 76c386bef61a4bd5a907266a5118ffe2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 13:56:49.664001 22913 log.cc:1196] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 76c386bef61a4bd5a907266a5118ffe2: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/wal/wals/7bdb9c14caf04abca39e56f8bcf1d1d4
I20240503 13:56:49.664664 12899 catalog_manager.cc:4830] TS 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885): tablet 7bdb9c14caf04abca39e56f8bcf1d1d4 (table rebalancer_extra_table_2 [id=65fb8aeaab8a4eb884e47e4e0f7ef87d]) successfully deleted
I20240503 13:56:49.706321 15493 raft_consensus.cc:1203] T b34b432bb82e465f87a6a3b05d7adfd2 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:56:49.713744 12910 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:56:49.719866 23031 tablet.cc:1722] T 6be43f8163a043f199d87ec2f969745a P 61b887f0d40d497badeca5604c1f0546: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:49.720196 23031 tablet.cc:1620] T 6be43f8163a043f199d87ec2f969745a P 61b887f0d40d497badeca5604c1f0546: MemRowSet was empty: no flush needed.
I20240503 13:56:49.720736 23032 tablet.cc:1722] T 6be43f8163a043f199d87ec2f969745a P e47105a0f46e4511b7112e571688a4c7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:49.721022 23032 tablet.cc:1620] T 6be43f8163a043f199d87ec2f969745a P e47105a0f46e4511b7112e571688a4c7: MemRowSet was empty: no flush needed.
I20240503 13:56:49.721267 23033 tablet.cc:1722] T 6be43f8163a043f199d87ec2f969745a P 7690ee79577c45e3a2c8f8b552c0bff8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:49.721534 23033 tablet.cc:1620] T 6be43f8163a043f199d87ec2f969745a P 7690ee79577c45e3a2c8f8b552c0bff8: MemRowSet was empty: no flush needed.
I20240503 13:56:49.728616 12909 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=53f10ccab76d45c3a497fed6e684bef5] alter complete (version 2)
I20240503 13:56:49.753559 23030 raft_consensus.cc:482] T 24b7efa8295b42668dc31a8083272924 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:49.753691 23030 raft_consensus.cc:504] T 24b7efa8295b42668dc31a8083272924 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:49.754129 23030 leader_election.cc:288] T 24b7efa8295b42668dc31a8083272924 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153), 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:49.754650 15493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24b7efa8295b42668dc31a8083272924" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546" is_pre_election: true
I20240503 13:56:49.754818 15493 raft_consensus.cc:2379] T 24b7efa8295b42668dc31a8083272924 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 76c386bef61a4bd5a907266a5118ffe2 in current term 1: Already voted for candidate 60ba3ffe3b7746469834cd91c84be14e in this term.
I20240503 13:56:49.755898 13051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24b7efa8295b42668dc31a8083272924" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e" is_pre_election: true
I20240503 13:56:49.756593 15692 leader_election.cc:302] T 24b7efa8295b42668dc31a8083272924 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76c386bef61a4bd5a907266a5118ffe2; no voters: 60ba3ffe3b7746469834cd91c84be14e, 61b887f0d40d497badeca5604c1f0546
I20240503 13:56:49.757755 23030 raft_consensus.cc:3037] T 24b7efa8295b42668dc31a8083272924 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:49.759002 23030 raft_consensus.cc:2726] T 24b7efa8295b42668dc31a8083272924 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 13:56:49.797411 15605 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:56:49.798954 15867 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:56:49.800707 15472 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 13:56:49.805773 15736 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:56:49.816116 13032 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 13:56:49.822292 13423 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:49.825865 13296 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 13:56:49.831808 12912 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:56:49.838021 13164 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 13:56:49.838958 23031 tablet.cc:1722] T 6be43f8163a043f199d87ec2f969745a P 61b887f0d40d497badeca5604c1f0546: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:49.842375 23033 tablet.cc:1722] T 6be43f8163a043f199d87ec2f969745a P 7690ee79577c45e3a2c8f8b552c0bff8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:49.847903 12905 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table_new_name [id=53f10ccab76d45c3a497fed6e684bef5] alter complete (version 3)
I20240503 13:56:49.848068 23032 tablet.cc:1722] T 6be43f8163a043f199d87ec2f969745a P e47105a0f46e4511b7112e571688a4c7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:49.913882 15889 raft_consensus.cc:1203] T 604875eff3594876af54e9126057bd7f P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:56:49.937106 13183 consensus_queue.cc:237] T a6a2c05fe3574ab98fb755935b6a92e1 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:49.938617 13051 raft_consensus.cc:1261] T a6a2c05fe3574ab98fb755935b6a92e1 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Refusing update from remote peer 8ccf5a4ac450414c9d3579bd5800b770: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:56:49.938844 13314 raft_consensus.cc:1261] T a6a2c05fe3574ab98fb755935b6a92e1 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Refusing update from remote peer 8ccf5a4ac450414c9d3579bd5800b770: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:56:49.938977 23143 consensus_queue.cc:1035] T a6a2c05fe3574ab98fb755935b6a92e1 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:49.939213 22421 consensus_queue.cc:1035] T a6a2c05fe3574ab98fb755935b6a92e1 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 13:56:49.939731 13121 consensus_peers.cc:485] T a6a2c05fe3574ab98fb755935b6a92e1 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153): Couldn't send request to peer 61b887f0d40d497badeca5604c1f0546. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a6a2c05fe3574ab98fb755935b6a92e1. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:49.941385 23025 raft_consensus.cc:2932] T a6a2c05fe3574ab98fb755935b6a92e1 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 61b887f0d40d497badeca5604c1f0546 (127.6.98.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:49.941526 23146 consensus_queue.cc:1035] T 17ef7be98ce84be2a0a8ac921a0e479d P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:49.941705 13314 raft_consensus.cc:2932] T a6a2c05fe3574ab98fb755935b6a92e1 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 61b887f0d40d497badeca5604c1f0546 (127.6.98.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:49.944856 12905 catalog_manager.cc:5484] T a6a2c05fe3574ab98fb755935b6a92e1 P ba0194098aa84449ae17e8a0d4c43703 reported cstate change: config changed from index -1 to 4, VOTER 61b887f0d40d497badeca5604c1f0546 (127.6.98.69) added. New cstate: current_term: 1 leader_uuid: "8ccf5a4ac450414c9d3579bd5800b770" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:49.946208 13051 raft_consensus.cc:2932] T a6a2c05fe3574ab98fb755935b6a92e1 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 61b887f0d40d497badeca5604c1f0546 (127.6.98.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:49.948554 22738 consensus_queue.cc:1035] T 013d4904c1ce48b78884dd61d9dacb32 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:49.951632 23146 consensus_queue.cc:1035] T 17ef7be98ce84be2a0a8ac921a0e479d P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:49.952945 20870 consensus_queue.cc:1035] T 013d4904c1ce48b78884dd61d9dacb32 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:49.953243 12909 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:49.953368 12909 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:49.957036 12909 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Sending DeleteTablet for 3 replicas of tablet 6be43f8163a043f199d87ec2f969745a
I20240503 13:56:49.957598 13423 tablet_service.cc:1514] Processing DeleteTablet for tablet 6be43f8163a043f199d87ec2f969745a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:49 UTC) from {username='slave'} at 127.0.0.1:33678
I20240503 13:56:49.957604 15867 tablet_service.cc:1514] Processing DeleteTablet for tablet 6be43f8163a043f199d87ec2f969745a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:49 UTC) from {username='slave'} at 127.0.0.1:50498
I20240503 13:56:49.957859 15472 tablet_service.cc:1514] Processing DeleteTablet for tablet 6be43f8163a043f199d87ec2f969745a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:49 UTC) from {username='slave'} at 127.0.0.1:58758
I20240503 13:56:49.957962 23159 tablet_replica.cc:330] T 6be43f8163a043f199d87ec2f969745a P e47105a0f46e4511b7112e571688a4c7: stopping tablet replica
I20240503 13:56:49.958102 23159 raft_consensus.cc:2229] T 6be43f8163a043f199d87ec2f969745a P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:49.958156 23160 tablet_replica.cc:330] T 6be43f8163a043f199d87ec2f969745a P 61b887f0d40d497badeca5604c1f0546: stopping tablet replica
I20240503 13:56:49.958262 23159 raft_consensus.cc:2258] T 6be43f8163a043f199d87ec2f969745a P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:49.958294 23160 raft_consensus.cc:2229] T 6be43f8163a043f199d87ec2f969745a P 61b887f0d40d497badeca5604c1f0546 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:56:49.958495 23160 raft_consensus.cc:2258] T 6be43f8163a043f199d87ec2f969745a P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:49.960081 23160 ts_tablet_manager.cc:1875] T 6be43f8163a043f199d87ec2f969745a P 61b887f0d40d497badeca5604c1f0546: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:49.961663 23158 tablet_replica.cc:330] T 6be43f8163a043f199d87ec2f969745a P 7690ee79577c45e3a2c8f8b552c0bff8: stopping tablet replica
I20240503 13:56:49.961773 23159 ts_tablet_manager.cc:1875] T 6be43f8163a043f199d87ec2f969745a P e47105a0f46e4511b7112e571688a4c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:49.961861 23158 raft_consensus.cc:2229] T 6be43f8163a043f199d87ec2f969745a P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:49.962030 23158 raft_consensus.cc:2258] T 6be43f8163a043f199d87ec2f969745a P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:49.962782 23158 ts_tablet_manager.cc:1875] T 6be43f8163a043f199d87ec2f969745a P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:49.963946 23160 ts_tablet_manager.cc:1888] T 6be43f8163a043f199d87ec2f969745a P 61b887f0d40d497badeca5604c1f0546: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:49.964586 23160 log.cc:1196] T 6be43f8163a043f199d87ec2f969745a P 61b887f0d40d497badeca5604c1f0546: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/wal/wals/6be43f8163a043f199d87ec2f969745a
I20240503 13:56:49.965276 23160 ts_tablet_manager.cc:1909] T 6be43f8163a043f199d87ec2f969745a P 61b887f0d40d497badeca5604c1f0546: Deleting consensus metadata
I20240503 13:56:49.966511 23158 ts_tablet_manager.cc:1888] T 6be43f8163a043f199d87ec2f969745a P 7690ee79577c45e3a2c8f8b552c0bff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:49.966665 23158 log.cc:1196] T 6be43f8163a043f199d87ec2f969745a P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal/wals/6be43f8163a043f199d87ec2f969745a
I20240503 13:56:49.967058 23158 ts_tablet_manager.cc:1909] T 6be43f8163a043f199d87ec2f969745a P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting consensus metadata
I20240503 13:56:49.967254 12899 catalog_manager.cc:4830] TS 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153): tablet 6be43f8163a043f199d87ec2f969745a (table rebalancer_dynamic_table_new_name [id=53f10ccab76d45c3a497fed6e684bef5]) successfully deleted
I20240503 13:56:49.967336 23159 ts_tablet_manager.cc:1888] T 6be43f8163a043f199d87ec2f969745a P e47105a0f46e4511b7112e571688a4c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:49.967458 23159 log.cc:1196] T 6be43f8163a043f199d87ec2f969745a P e47105a0f46e4511b7112e571688a4c7: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-7/wal/wals/6be43f8163a043f199d87ec2f969745a
I20240503 13:56:49.967828 12896 catalog_manager.cc:4830] TS 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109): tablet 6be43f8163a043f199d87ec2f969745a (table rebalancer_dynamic_table_new_name [id=53f10ccab76d45c3a497fed6e684bef5]) successfully deleted
I20240503 13:56:49.967936 23159 ts_tablet_manager.cc:1909] T 6be43f8163a043f199d87ec2f969745a P e47105a0f46e4511b7112e571688a4c7: Deleting consensus metadata
I20240503 13:56:49.968817 12896 catalog_manager.cc:4830] TS e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): tablet 6be43f8163a043f199d87ec2f969745a (table rebalancer_dynamic_table_new_name [id=53f10ccab76d45c3a497fed6e684bef5]) successfully deleted
I20240503 13:56:49.973013 22738 consensus_queue.cc:1035] T 24b7efa8295b42668dc31a8083272924 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:49.976751 20870 consensus_queue.cc:1035] T 24b7efa8295b42668dc31a8083272924 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:49.986308 13314 consensus_queue.cc:237] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:49.987593 13183 raft_consensus.cc:1261] T 866723d26350414197ec27544277db9c P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:56:49.987891 13447 raft_consensus.cc:1261] T 866723d26350414197ec27544277db9c P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 13:56:49.988071 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 866723d26350414197ec27544277db9c. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:49.988122 23002 consensus_queue.cc:1035] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:49.988288 22614 consensus_queue.cc:1035] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:49.990437 22614 raft_consensus.cc:2932] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:49.990881 13183 raft_consensus.cc:2932] T 866723d26350414197ec27544277db9c P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:49.992697 13447 raft_consensus.cc:2932] T 866723d26350414197ec27544277db9c P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:49.994149 12905 catalog_manager.cc:5484] T 866723d26350414197ec27544277db9c P 8ccf5a4ac450414c9d3579bd5800b770 reported cstate change: config changed from index -1 to 4, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) added. New cstate: current_term: 1 leader_uuid: "ba0194098aa84449ae17e8a0d4c43703" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:50.015960 15889 consensus_queue.cc:237] T 3a140864f59b483bb5fea3de83d28ecf P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:50.016376 15888 raft_consensus.cc:1203] T c7d5e8c09ec14ac7891c8a63cb896878 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240503 13:56:50.017571 15825 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P e47105a0f46e4511b7112e571688a4c7 -> Peer ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279): Couldn't send request to peer ba0194098aa84449ae17e8a0d4c43703. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3a140864f59b483bb5fea3de83d28ecf. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:50.017779 15624 raft_consensus.cc:1261] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Refusing update from remote peer e47105a0f46e4511b7112e571688a4c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:50.018193 22256 consensus_queue.cc:1035] T 3a140864f59b483bb5fea3de83d28ecf P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:56:50.018896 15758 raft_consensus.cc:1261] T 3a140864f59b483bb5fea3de83d28ecf P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Refusing update from remote peer e47105a0f46e4511b7112e571688a4c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:50.019261 22256 consensus_queue.cc:1035] T 3a140864f59b483bb5fea3de83d28ecf P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:56:50.021193 22630 raft_consensus.cc:2932] T 3a140864f59b483bb5fea3de83d28ecf P e47105a0f46e4511b7112e571688a4c7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:50.021662 15758 raft_consensus.cc:2932] T 3a140864f59b483bb5fea3de83d28ecf P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:50.023241 15624 raft_consensus.cc:2932] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:50.024645 12909 catalog_manager.cc:5484] T 3a140864f59b483bb5fea3de83d28ecf P e47105a0f46e4511b7112e571688a4c7 reported cstate change: config changed from index -1 to 2, VOTER ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) added. New cstate: current_term: 1 leader_uuid: "e47105a0f46e4511b7112e571688a4c7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:50.047061 13314 consensus_queue.cc:237] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:50.048547 13183 raft_consensus.cc:1261] T 84f1ab11e74045f8a9d7033c5ead0c14 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:56:50.048712 13447 raft_consensus.cc:1261] T 84f1ab11e74045f8a9d7033c5ead0c14 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 13:56:50.048841 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 84f1ab11e74045f8a9d7033c5ead0c14. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:50.049010 23002 consensus_queue.cc:1035] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:50.049185 22614 consensus_queue.cc:1035] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:50.051084 23002 raft_consensus.cc:2932] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:50.051559 13183 raft_consensus.cc:2932] T 84f1ab11e74045f8a9d7033c5ead0c14 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:50.053382 13447 raft_consensus.cc:2932] T 84f1ab11e74045f8a9d7033c5ead0c14 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:50.054661 12907 catalog_manager.cc:5484] T 84f1ab11e74045f8a9d7033c5ead0c14 P 8ccf5a4ac450414c9d3579bd5800b770 reported cstate change: config changed from index -1 to 4, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) added. New cstate: current_term: 1 leader_uuid: "ba0194098aa84449ae17e8a0d4c43703" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:50.058966 12905 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:56:50.065145 13164 tablet_service.cc:1467] Processing CreateTablet for tablet a39982c8458f4291bf292b3436ad8a0a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=906e52f86b624a34ab9ad0faaf2d8757]), partition=
I20240503 13:56:50.065171 13423 tablet_service.cc:1467] Processing CreateTablet for tablet a39982c8458f4291bf292b3436ad8a0a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=906e52f86b624a34ab9ad0faaf2d8757]), partition=
I20240503 13:56:50.065420 13164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a39982c8458f4291bf292b3436ad8a0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:50.065420 13423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a39982c8458f4291bf292b3436ad8a0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:50.066352 15867 tablet_service.cc:1467] Processing CreateTablet for tablet a39982c8458f4291bf292b3436ad8a0a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=906e52f86b624a34ab9ad0faaf2d8757]), partition=
I20240503 13:56:50.066598 15867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a39982c8458f4291bf292b3436ad8a0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:50.067411 23227 tablet_bootstrap.cc:492] T a39982c8458f4291bf292b3436ad8a0a P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:50.069177 23229 tablet_bootstrap.cc:492] T a39982c8458f4291bf292b3436ad8a0a P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:50.069509 23228 tablet_bootstrap.cc:492] T a39982c8458f4291bf292b3436ad8a0a P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:50.070194 23229 tablet_bootstrap.cc:654] T a39982c8458f4291bf292b3436ad8a0a P e47105a0f46e4511b7112e571688a4c7: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:50.070474 23228 tablet_bootstrap.cc:654] T a39982c8458f4291bf292b3436ad8a0a P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:50.071321 23229 tablet_bootstrap.cc:492] T a39982c8458f4291bf292b3436ad8a0a P e47105a0f46e4511b7112e571688a4c7: No bootstrap required, opened a new log
I20240503 13:56:50.071406 23229 ts_tablet_manager.cc:1367] T a39982c8458f4291bf292b3436ad8a0a P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:56:50.071516 23228 tablet_bootstrap.cc:492] T a39982c8458f4291bf292b3436ad8a0a P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:50.071591 23228 ts_tablet_manager.cc:1367] T a39982c8458f4291bf292b3436ad8a0a P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:50.071892 23229 raft_consensus.cc:348] T a39982c8458f4291bf292b3436ad8a0a P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:50.071998 23229 raft_consensus.cc:374] T a39982c8458f4291bf292b3436ad8a0a P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:50.071961 23228 raft_consensus.cc:348] T a39982c8458f4291bf292b3436ad8a0a P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:50.072028 23229 raft_consensus.cc:729] T a39982c8458f4291bf292b3436ad8a0a P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:50.072059 23228 raft_consensus.cc:374] T a39982c8458f4291bf292b3436ad8a0a P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:50.072086 23228 raft_consensus.cc:729] T a39982c8458f4291bf292b3436ad8a0a P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:50.072146 23229 consensus_queue.cc:260] T a39982c8458f4291bf292b3436ad8a0a P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:50.072194 23228 consensus_queue.cc:260] T a39982c8458f4291bf292b3436ad8a0a P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:50.072418 23229 ts_tablet_manager.cc:1398] T a39982c8458f4291bf292b3436ad8a0a P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:50.072435 23228 ts_tablet_manager.cc:1398] T a39982c8458f4291bf292b3436ad8a0a P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:50.072996 23227 tablet_bootstrap.cc:654] T a39982c8458f4291bf292b3436ad8a0a P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:50.076085 23227 tablet_bootstrap.cc:492] T a39982c8458f4291bf292b3436ad8a0a P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:50.076196 23227 ts_tablet_manager.cc:1367] T a39982c8458f4291bf292b3436ad8a0a P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 13:56:50.077787 23227 raft_consensus.cc:348] T a39982c8458f4291bf292b3436ad8a0a P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:50.078083 23227 raft_consensus.cc:374] T a39982c8458f4291bf292b3436ad8a0a P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:50.078241 23227 raft_consensus.cc:729] T a39982c8458f4291bf292b3436ad8a0a P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:50.078536 23227 consensus_queue.cc:260] T a39982c8458f4291bf292b3436ad8a0a P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:50.079572 23227 ts_tablet_manager.cc:1398] T a39982c8458f4291bf292b3436ad8a0a P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 13:56:50.082350 12905 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_extra_table_54"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:56:50.089649 15472 tablet_service.cc:1467] Processing CreateTablet for tablet ada3e4b682c043d585d851c01f689a5c (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=8280434cc9ad490ebe9370f6490250d5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:50.089879 13296 tablet_service.cc:1467] Processing CreateTablet for tablet ada3e4b682c043d585d851c01f689a5c (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=8280434cc9ad490ebe9370f6490250d5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:50.089978 15472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ada3e4b682c043d585d851c01f689a5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:50.090194 13296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ada3e4b682c043d585d851c01f689a5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:50.092222 23239 tablet_bootstrap.cc:492] T ada3e4b682c043d585d851c01f689a5c P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:50.093099 15605 tablet_service.cc:1467] Processing CreateTablet for tablet ada3e4b682c043d585d851c01f689a5c (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=8280434cc9ad490ebe9370f6490250d5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:50.093372 15605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ada3e4b682c043d585d851c01f689a5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:50.093349 13032 tablet_service.cc:1467] Processing CreateTablet for tablet a3418454434b4a9791d50022b8d35743 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=8280434cc9ad490ebe9370f6490250d5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:50.093611 13032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3418454434b4a9791d50022b8d35743. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:50.093788 23239 tablet_bootstrap.cc:654] T ada3e4b682c043d585d851c01f689a5c P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:50.094905 23240 tablet_bootstrap.cc:492] T ada3e4b682c043d585d851c01f689a5c P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:50.094982 23239 tablet_bootstrap.cc:492] T ada3e4b682c043d585d851c01f689a5c P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:50.095054 23239 ts_tablet_manager.cc:1367] T ada3e4b682c043d585d851c01f689a5c P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:56:50.095479 23239 raft_consensus.cc:348] T ada3e4b682c043d585d851c01f689a5c P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:50.095594 23239 raft_consensus.cc:374] T ada3e4b682c043d585d851c01f689a5c P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:50.095625 23239 raft_consensus.cc:729] T ada3e4b682c043d585d851c01f689a5c P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:50.095744 23239 consensus_queue.cc:260] T ada3e4b682c043d585d851c01f689a5c P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:50.095860 15604 tablet_service.cc:1467] Processing CreateTablet for tablet a3418454434b4a9791d50022b8d35743 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=8280434cc9ad490ebe9370f6490250d5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:50.096024 23239 ts_tablet_manager.cc:1398] T ada3e4b682c043d585d851c01f689a5c P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:50.096031 13423 tablet_service.cc:1467] Processing CreateTablet for tablet a3418454434b4a9791d50022b8d35743 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=8280434cc9ad490ebe9370f6490250d5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:50.096105 15604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3418454434b4a9791d50022b8d35743. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:50.096132 23240 tablet_bootstrap.cc:654] T ada3e4b682c043d585d851c01f689a5c P 61b887f0d40d497badeca5604c1f0546: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:50.096276 13423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3418454434b4a9791d50022b8d35743. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:50.097049 15867 tablet_service.cc:1467] Processing CreateTablet for tablet a2b9dcff1b72490397f2afc5348923c1 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=8280434cc9ad490ebe9370f6490250d5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:50.097304 15867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2b9dcff1b72490397f2afc5348923c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:50.098069 23240 tablet_bootstrap.cc:492] T ada3e4b682c043d585d851c01f689a5c P 61b887f0d40d497badeca5604c1f0546: No bootstrap required, opened a new log
I20240503 13:56:50.098152 23240 ts_tablet_manager.cc:1367] T ada3e4b682c043d585d851c01f689a5c P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 13:56:50.098806 23240 raft_consensus.cc:348] T ada3e4b682c043d585d851c01f689a5c P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:50.098891 23240 raft_consensus.cc:374] T ada3e4b682c043d585d851c01f689a5c P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:50.098910 23240 raft_consensus.cc:729] T ada3e4b682c043d585d851c01f689a5c P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:50.099017 23240 consensus_queue.cc:260] T ada3e4b682c043d585d851c01f689a5c P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:50.099277 23240 ts_tablet_manager.cc:1398] T ada3e4b682c043d585d851c01f689a5c P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:50.100688 23242 tablet_bootstrap.cc:492] T ada3e4b682c043d585d851c01f689a5c P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:50.100649 15472 tablet_service.cc:1467] Processing CreateTablet for tablet a2b9dcff1b72490397f2afc5348923c1 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=8280434cc9ad490ebe9370f6490250d5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:50.100890 15472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2b9dcff1b72490397f2afc5348923c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:50.101172 15736 tablet_service.cc:1467] Processing CreateTablet for tablet a2b9dcff1b72490397f2afc5348923c1 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=8280434cc9ad490ebe9370f6490250d5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:50.101368 23243 tablet_bootstrap.cc:492] T a3418454434b4a9791d50022b8d35743 P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:50.101441 15736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2b9dcff1b72490397f2afc5348923c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:50.102500 23242 tablet_bootstrap.cc:654] T ada3e4b682c043d585d851c01f689a5c P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:50.102938 23240 tablet_bootstrap.cc:492] T a2b9dcff1b72490397f2afc5348923c1 P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:50.103623 23245 tablet_bootstrap.cc:492] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:50.103801 23240 tablet_bootstrap.cc:654] T a2b9dcff1b72490397f2afc5348923c1 P 61b887f0d40d497badeca5604c1f0546: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:50.104535 23243 tablet_bootstrap.cc:654] T a3418454434b4a9791d50022b8d35743 P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:50.104986 23240 tablet_bootstrap.cc:492] T a2b9dcff1b72490397f2afc5348923c1 P 61b887f0d40d497badeca5604c1f0546: No bootstrap required, opened a new log
I20240503 13:56:50.105053 23240 ts_tablet_manager.cc:1367] T a2b9dcff1b72490397f2afc5348923c1 P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 13:56:50.105434 23240 raft_consensus.cc:348] T a2b9dcff1b72490397f2afc5348923c1 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:50.105542 23240 raft_consensus.cc:374] T a2b9dcff1b72490397f2afc5348923c1 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:50.105571 23240 raft_consensus.cc:729] T a2b9dcff1b72490397f2afc5348923c1 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:50.105674 23240 consensus_queue.cc:260] T a2b9dcff1b72490397f2afc5348923c1 P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:50.105926 23240 ts_tablet_manager.cc:1398] T a2b9dcff1b72490397f2afc5348923c1 P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:50.108371 23245 tablet_bootstrap.cc:654] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:50.108661 23229 tablet_bootstrap.cc:492] T a2b9dcff1b72490397f2afc5348923c1 P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:50.108752 23228 tablet_bootstrap.cc:492] T a3418454434b4a9791d50022b8d35743 P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:50.109807 23228 tablet_bootstrap.cc:654] T a3418454434b4a9791d50022b8d35743 P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:50.109822 23229 tablet_bootstrap.cc:654] T a2b9dcff1b72490397f2afc5348923c1 P e47105a0f46e4511b7112e571688a4c7: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:50.110630 23242 tablet_bootstrap.cc:492] T ada3e4b682c043d585d851c01f689a5c P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:50.110713 23242 ts_tablet_manager.cc:1367] T ada3e4b682c043d585d851c01f689a5c P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 13:56:50.113282 23242 raft_consensus.cc:348] T ada3e4b682c043d585d851c01f689a5c P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:50.113592 23243 tablet_bootstrap.cc:492] T a3418454434b4a9791d50022b8d35743 P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:50.113734 23243 ts_tablet_manager.cc:1367] T a3418454434b4a9791d50022b8d35743 P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 13:56:50.113830 23242 raft_consensus.cc:374] T ada3e4b682c043d585d851c01f689a5c P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:50.114691 23242 raft_consensus.cc:729] T ada3e4b682c043d585d851c01f689a5c P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:50.114895 23242 consensus_queue.cc:260] T ada3e4b682c043d585d851c01f689a5c P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:50.115279 23243 raft_consensus.cc:348] T a3418454434b4a9791d50022b8d35743 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:50.115643 23242 ts_tablet_manager.cc:1398] T ada3e4b682c043d585d851c01f689a5c P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240503 13:56:50.115756 23243 raft_consensus.cc:374] T a3418454434b4a9791d50022b8d35743 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:50.115849 23243 raft_consensus.cc:729] T a3418454434b4a9791d50022b8d35743 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:50.115911 23242 tablet_bootstrap.cc:492] T a3418454434b4a9791d50022b8d35743 P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:50.116746 23243 consensus_queue.cc:260] T a3418454434b4a9791d50022b8d35743 P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:50.117169 23242 tablet_bootstrap.cc:654] T a3418454434b4a9791d50022b8d35743 P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:50.117255 23243 ts_tablet_manager.cc:1398] T a3418454434b4a9791d50022b8d35743 P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:56:50.117945 23245 tablet_bootstrap.cc:492] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:50.118142 23228 tablet_bootstrap.cc:492] T a3418454434b4a9791d50022b8d35743 P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:50.118242 23228 ts_tablet_manager.cc:1367] T a3418454434b4a9791d50022b8d35743 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 13:56:50.120396 23228 raft_consensus.cc:348] T a3418454434b4a9791d50022b8d35743 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:50.120553 23228 raft_consensus.cc:374] T a3418454434b4a9791d50022b8d35743 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:50.120590 23228 raft_consensus.cc:729] T a3418454434b4a9791d50022b8d35743 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:50.120709 23228 consensus_queue.cc:260] T a3418454434b4a9791d50022b8d35743 P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:50.121006 23228 ts_tablet_manager.cc:1398] T a3418454434b4a9791d50022b8d35743 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:56:50.124848 23245 ts_tablet_manager.cc:1367] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.000s
I20240503 13:56:50.126190 23245 raft_consensus.cc:348] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:50.126333 23245 raft_consensus.cc:374] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:50.126368 23245 raft_consensus.cc:729] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:50.126487 23245 consensus_queue.cc:260] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:50.126776 23245 ts_tablet_manager.cc:1398] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:50.128085 23242 tablet_bootstrap.cc:492] T a3418454434b4a9791d50022b8d35743 P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:50.128221 23242 ts_tablet_manager.cc:1367] T a3418454434b4a9791d50022b8d35743 P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 13:56:50.128743 23242 raft_consensus.cc:348] T a3418454434b4a9791d50022b8d35743 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:50.128878 23242 raft_consensus.cc:374] T a3418454434b4a9791d50022b8d35743 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:50.128902 23242 raft_consensus.cc:729] T a3418454434b4a9791d50022b8d35743 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:50.129019 23242 consensus_queue.cc:260] T a3418454434b4a9791d50022b8d35743 P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:50.129297 23242 ts_tablet_manager.cc:1398] T a3418454434b4a9791d50022b8d35743 P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:50.130159 23229 tablet_bootstrap.cc:492] T a2b9dcff1b72490397f2afc5348923c1 P e47105a0f46e4511b7112e571688a4c7: No bootstrap required, opened a new log
I20240503 13:56:50.130255 23229 ts_tablet_manager.cc:1367] T a2b9dcff1b72490397f2afc5348923c1 P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20240503 13:56:50.130694 23229 raft_consensus.cc:348] T a2b9dcff1b72490397f2afc5348923c1 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:50.130789 23229 raft_consensus.cc:374] T a2b9dcff1b72490397f2afc5348923c1 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:50.130812 23229 raft_consensus.cc:729] T a2b9dcff1b72490397f2afc5348923c1 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:50.130932 23229 consensus_queue.cc:260] T a2b9dcff1b72490397f2afc5348923c1 P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:50.131220 23229 ts_tablet_manager.cc:1398] T a2b9dcff1b72490397f2afc5348923c1 P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:50.144891 23094 raft_consensus.cc:482] T ada3e4b682c043d585d851c01f689a5c P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:50.145006 23094 raft_consensus.cc:504] T ada3e4b682c043d585d851c01f689a5c P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:50.145385 23094 leader_election.cc:288] T ada3e4b682c043d585d851c01f689a5c P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951), ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:50.145831 15624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada3e4b682c043d585d851c01f689a5c" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d" is_pre_election: true
I20240503 13:56:50.145972 15624 raft_consensus.cc:2454] T ada3e4b682c043d585d851c01f689a5c P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 0.
I20240503 13:56:50.145959 13316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada3e4b682c043d585d851c01f689a5c" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703" is_pre_election: true
I20240503 13:56:50.146106 13316 raft_consensus.cc:2454] T ada3e4b682c043d585d851c01f689a5c P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 0.
I20240503 13:56:50.146472 15429 leader_election.cc:302] T ada3e4b682c043d585d851c01f689a5c P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61b887f0d40d497badeca5604c1f0546, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:50.146747 23094 raft_consensus.cc:2781] T ada3e4b682c043d585d851c01f689a5c P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:50.146802 23094 raft_consensus.cc:482] T ada3e4b682c043d585d851c01f689a5c P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:50.146835 23094 raft_consensus.cc:3037] T ada3e4b682c043d585d851c01f689a5c P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:50.147980 23094 raft_consensus.cc:504] T ada3e4b682c043d585d851c01f689a5c P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:50.148319 23094 leader_election.cc:288] T ada3e4b682c043d585d851c01f689a5c P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 election: Requested vote from peers 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951), ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:50.150332 13316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada3e4b682c043d585d851c01f689a5c" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703"
I20240503 13:56:50.150457 13316 raft_consensus.cc:3037] T ada3e4b682c043d585d851c01f689a5c P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:50.151537 13316 raft_consensus.cc:2454] T ada3e4b682c043d585d851c01f689a5c P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 1.
I20240503 13:56:50.152175 15624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada3e4b682c043d585d851c01f689a5c" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d"
I20240503 13:56:50.152289 15624 raft_consensus.cc:3037] T ada3e4b682c043d585d851c01f689a5c P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:50.153347 15624 raft_consensus.cc:2454] T ada3e4b682c043d585d851c01f689a5c P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 1.
I20240503 13:56:50.153689 15426 leader_election.cc:302] T ada3e4b682c043d585d851c01f689a5c P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 61b887f0d40d497badeca5604c1f0546; no voters:
I20240503 13:56:50.154155 21492 raft_consensus.cc:2781] T ada3e4b682c043d585d851c01f689a5c P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:50.154249 21492 raft_consensus.cc:686] T ada3e4b682c043d585d851c01f689a5c P 61b887f0d40d497badeca5604c1f0546 [term 1 LEADER]: Becoming Leader. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Running, Role: LEADER
I20240503 13:56:50.154389 21492 consensus_queue.cc:237] T ada3e4b682c043d585d851c01f689a5c P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:50.156934 12905 catalog_manager.cc:5484] T ada3e4b682c043d585d851c01f689a5c P 61b887f0d40d497badeca5604c1f0546 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61b887f0d40d497badeca5604c1f0546 (127.6.98.69). New cstate: current_term: 1 leader_uuid: "61b887f0d40d497badeca5604c1f0546" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:50.220389 23172 raft_consensus.cc:482] T a39982c8458f4291bf292b3436ad8a0a P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:50.220557 23172 raft_consensus.cc:504] T a39982c8458f4291bf292b3436ad8a0a P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:50.221119 23172 leader_election.cc:288] T a39982c8458f4291bf292b3436ad8a0a P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:50.221423 13183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39982c8458f4291bf292b3436ad8a0a" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770" is_pre_election: true
I20240503 13:56:50.221426 15889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39982c8458f4291bf292b3436ad8a0a" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7" is_pre_election: true
I20240503 13:56:50.221556 15889 raft_consensus.cc:2454] T a39982c8458f4291bf292b3436ad8a0a P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 0.
I20240503 13:56:50.221566 13183 raft_consensus.cc:2454] T a39982c8458f4291bf292b3436ad8a0a P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 0.
I20240503 13:56:50.221860 13380 leader_election.cc:302] T a39982c8458f4291bf292b3436ad8a0a P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:50.222057 23172 raft_consensus.cc:2781] T a39982c8458f4291bf292b3436ad8a0a P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:50.222115 23172 raft_consensus.cc:482] T a39982c8458f4291bf292b3436ad8a0a P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:50.222141 23172 raft_consensus.cc:3037] T a39982c8458f4291bf292b3436ad8a0a P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:50.223106 23172 raft_consensus.cc:504] T a39982c8458f4291bf292b3436ad8a0a P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:50.223408 23172 leader_election.cc:288] T a39982c8458f4291bf292b3436ad8a0a P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 election: Requested vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:50.223701 15889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39982c8458f4291bf292b3436ad8a0a" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7"
I20240503 13:56:50.223755 13183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39982c8458f4291bf292b3436ad8a0a" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770"
I20240503 13:56:50.223834 15889 raft_consensus.cc:3037] T a39982c8458f4291bf292b3436ad8a0a P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:50.223850 13183 raft_consensus.cc:3037] T a39982c8458f4291bf292b3436ad8a0a P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:50.224859 13183 raft_consensus.cc:2454] T a39982c8458f4291bf292b3436ad8a0a P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 1.
I20240503 13:56:50.224859 15889 raft_consensus.cc:2454] T a39982c8458f4291bf292b3436ad8a0a P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 1.
I20240503 13:56:50.225108 13380 leader_election.cc:302] T a39982c8458f4291bf292b3436ad8a0a P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:50.225301 23172 raft_consensus.cc:2781] T a39982c8458f4291bf292b3436ad8a0a P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:50.225373 23172 raft_consensus.cc:686] T a39982c8458f4291bf292b3436ad8a0a P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 LEADER]: Becoming Leader. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Running, Role: LEADER
I20240503 13:56:50.225495 23172 consensus_queue.cc:237] T a39982c8458f4291bf292b3436ad8a0a P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:50.227859 12905 catalog_manager.cc:5484] T a39982c8458f4291bf292b3436ad8a0a P 7690ee79577c45e3a2c8f8b552c0bff8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68). New cstate: current_term: 1 leader_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } }
I20240503 13:56:50.295267 22738 raft_consensus.cc:482] T a3418454434b4a9791d50022b8d35743 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:50.295414 22738 raft_consensus.cc:504] T a3418454434b4a9791d50022b8d35743 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:50.295892 22738 leader_election.cc:288] T a3418454434b4a9791d50022b8d35743 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951)
I20240503 13:56:50.296304 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3418454434b4a9791d50022b8d35743" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
I20240503 13:56:50.296367 15624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3418454434b4a9791d50022b8d35743" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d" is_pre_election: true
I20240503 13:56:50.296458 13445 raft_consensus.cc:2454] T a3418454434b4a9791d50022b8d35743 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 0.
I20240503 13:56:50.296548 15624 raft_consensus.cc:2454] T a3418454434b4a9791d50022b8d35743 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 0.
I20240503 13:56:50.296890 12987 leader_election.cc:302] T a3418454434b4a9791d50022b8d35743 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 7690ee79577c45e3a2c8f8b552c0bff8; no voters:
I20240503 13:56:50.297081 22738 raft_consensus.cc:2781] T a3418454434b4a9791d50022b8d35743 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:50.297138 22738 raft_consensus.cc:482] T a3418454434b4a9791d50022b8d35743 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:50.297163 22738 raft_consensus.cc:3037] T a3418454434b4a9791d50022b8d35743 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:50.298091 22738 raft_consensus.cc:504] T a3418454434b4a9791d50022b8d35743 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:50.298465 22738 leader_election.cc:288] T a3418454434b4a9791d50022b8d35743 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 election: Requested vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951)
I20240503 13:56:50.298736 15624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3418454434b4a9791d50022b8d35743" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d"
I20240503 13:56:50.298738 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3418454434b4a9791d50022b8d35743" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
I20240503 13:56:50.298840 15624 raft_consensus.cc:3037] T a3418454434b4a9791d50022b8d35743 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:50.298877 13445 raft_consensus.cc:3037] T a3418454434b4a9791d50022b8d35743 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:50.299669 13445 raft_consensus.cc:2454] T a3418454434b4a9791d50022b8d35743 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 1.
I20240503 13:56:50.299700 15624 raft_consensus.cc:2454] T a3418454434b4a9791d50022b8d35743 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 1.
I20240503 13:56:50.299937 12987 leader_election.cc:302] T a3418454434b4a9791d50022b8d35743 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 7690ee79577c45e3a2c8f8b552c0bff8; no voters:
I20240503 13:56:50.300096 22738 raft_consensus.cc:2781] T a3418454434b4a9791d50022b8d35743 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:50.300264 22738 raft_consensus.cc:686] T a3418454434b4a9791d50022b8d35743 P 60ba3ffe3b7746469834cd91c84be14e [term 1 LEADER]: Becoming Leader. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Running, Role: LEADER
I20240503 13:56:50.300403 22738 consensus_queue.cc:237] T a3418454434b4a9791d50022b8d35743 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:50.302786 12912 catalog_manager.cc:5484] T a3418454434b4a9791d50022b8d35743 P 60ba3ffe3b7746469834cd91c84be14e reported cstate change: term changed from 0 to 1, leader changed from <none> to 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65). New cstate: current_term: 1 leader_uuid: "60ba3ffe3b7746469834cd91c84be14e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } }
I20240503 13:56:50.338143 12912 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:56:50.342717 13183 raft_consensus.cc:1261] T a39982c8458f4291bf292b3436ad8a0a P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Refusing update from remote peer 7690ee79577c45e3a2c8f8b552c0bff8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:50.342957 15889 raft_consensus.cc:1261] T a39982c8458f4291bf292b3436ad8a0a P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Refusing update from remote peer 7690ee79577c45e3a2c8f8b552c0bff8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:50.343295 22762 consensus_queue.cc:1035] T a39982c8458f4291bf292b3436ad8a0a P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:50.343421 23172 consensus_queue.cc:1035] T a39982c8458f4291bf292b3436ad8a0a P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:50.346338 12912 catalog_manager.cc:5576] TS 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387) does not have the latest schema for tablet a39982c8458f4291bf292b3436ad8a0a (table rebalancer_dynamic_table [id=906e52f86b624a34ab9ad0faaf2d8757]). Expected version 1 got 0
I20240503 13:56:50.346653 12905 catalog_manager.cc:5576] TS e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753) does not have the latest schema for tablet a39982c8458f4291bf292b3436ad8a0a (table rebalancer_dynamic_table [id=906e52f86b624a34ab9ad0faaf2d8757]). Expected version 1 got 0
I20240503 13:56:50.348686 23032 tablet.cc:1722] T a39982c8458f4291bf292b3436ad8a0a P e47105a0f46e4511b7112e571688a4c7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:50.348832 23293 tablet.cc:1722] T a39982c8458f4291bf292b3436ad8a0a P 8ccf5a4ac450414c9d3579bd5800b770: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:50.349030 23032 tablet.cc:1620] T a39982c8458f4291bf292b3436ad8a0a P e47105a0f46e4511b7112e571688a4c7: MemRowSet was empty: no flush needed.
I20240503 13:56:50.349546 23293 tablet.cc:1620] T a39982c8458f4291bf292b3436ad8a0a P 8ccf5a4ac450414c9d3579bd5800b770: MemRowSet was empty: no flush needed.
I20240503 13:56:50.349758 23292 tablet.cc:1722] T a39982c8458f4291bf292b3436ad8a0a P 7690ee79577c45e3a2c8f8b552c0bff8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:50.350456 12905 catalog_manager.cc:5576] TS 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387) does not have the latest schema for tablet a39982c8458f4291bf292b3436ad8a0a (table rebalancer_dynamic_table [id=906e52f86b624a34ab9ad0faaf2d8757]). Expected version 1 got 0
I20240503 13:56:50.350504 23292 tablet.cc:1620] T a39982c8458f4291bf292b3436ad8a0a P 7690ee79577c45e3a2c8f8b552c0bff8: MemRowSet was empty: no flush needed.
I20240503 13:56:50.352116 12912 catalog_manager.cc:5576] TS e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753) does not have the latest schema for tablet a39982c8458f4291bf292b3436ad8a0a (table rebalancer_dynamic_table [id=906e52f86b624a34ab9ad0faaf2d8757]). Expected version 1 got 0
I20240503 13:56:50.357038 12912 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=906e52f86b624a34ab9ad0faaf2d8757] alter complete (version 1)
I20240503 13:56:50.407223 23294 ts_tablet_manager.cc:902] T a6a2c05fe3574ab98fb755935b6a92e1 P 61b887f0d40d497badeca5604c1f0546: Initiating tablet copy from peer 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:50.407814 23294 tablet_copy_client.cc:308] T a6a2c05fe3574ab98fb755935b6a92e1 P 61b887f0d40d497badeca5604c1f0546: tablet copy: Beginning tablet copy session from remote peer at address 127.6.98.66:36387
I20240503 13:56:50.408272 23299 ts_tablet_manager.cc:902] T 3a140864f59b483bb5fea3de83d28ecf P ba0194098aa84449ae17e8a0d4c43703: Initiating tablet copy from peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:50.408463 13205 tablet_copy_service.cc:140] P 8ccf5a4ac450414c9d3579bd5800b770: Received BeginTabletCopySession request for tablet a6a2c05fe3574ab98fb755935b6a92e1 from peer 61b887f0d40d497badeca5604c1f0546 ({username='slave'} at 127.6.98.69:41857)
I20240503 13:56:50.408604 13205 tablet_copy_service.cc:161] P 8ccf5a4ac450414c9d3579bd5800b770: Beginning new tablet copy session on tablet a6a2c05fe3574ab98fb755935b6a92e1 from peer 61b887f0d40d497badeca5604c1f0546 at {username='slave'} at 127.6.98.69:41857: session id = 61b887f0d40d497badeca5604c1f0546-a6a2c05fe3574ab98fb755935b6a92e1
I20240503 13:56:50.408794 23299 tablet_copy_client.cc:308] T 3a140864f59b483bb5fea3de83d28ecf P ba0194098aa84449ae17e8a0d4c43703: tablet copy: Beginning tablet copy session from remote peer at address 127.6.98.72:44753
I20240503 13:56:50.409225 15909 tablet_copy_service.cc:140] P e47105a0f46e4511b7112e571688a4c7: Received BeginTabletCopySession request for tablet 3a140864f59b483bb5fea3de83d28ecf from peer ba0194098aa84449ae17e8a0d4c43703 ({username='slave'} at 127.6.98.67:56843)
I20240503 13:56:50.409309 15909 tablet_copy_service.cc:161] P e47105a0f46e4511b7112e571688a4c7: Beginning new tablet copy session on tablet 3a140864f59b483bb5fea3de83d28ecf from peer ba0194098aa84449ae17e8a0d4c43703 at {username='slave'} at 127.6.98.67:56843: session id = ba0194098aa84449ae17e8a0d4c43703-3a140864f59b483bb5fea3de83d28ecf
I20240503 13:56:50.409485 13205 tablet_copy_source_session.cc:205] T a6a2c05fe3574ab98fb755935b6a92e1 P 8ccf5a4ac450414c9d3579bd5800b770: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:56:50.410161 15909 tablet_copy_source_session.cc:205] T 3a140864f59b483bb5fea3de83d28ecf P e47105a0f46e4511b7112e571688a4c7: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:56:50.410331 23294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6a2c05fe3574ab98fb755935b6a92e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:50.410986 23299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a140864f59b483bb5fea3de83d28ecf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:50.412292 23294 tablet_copy_client.cc:783] T a6a2c05fe3574ab98fb755935b6a92e1 P 61b887f0d40d497badeca5604c1f0546: tablet copy: Starting download of 0 data blocks...
I20240503 13:56:50.412442 23294 tablet_copy_client.cc:647] T a6a2c05fe3574ab98fb755935b6a92e1 P 61b887f0d40d497badeca5604c1f0546: tablet copy: Starting download of 1 WAL segments...
I20240503 13:56:50.412912 23299 tablet_copy_client.cc:783] T 3a140864f59b483bb5fea3de83d28ecf P ba0194098aa84449ae17e8a0d4c43703: tablet copy: Starting download of 0 data blocks...
I20240503 13:56:50.413053 23299 tablet_copy_client.cc:647] T 3a140864f59b483bb5fea3de83d28ecf P ba0194098aa84449ae17e8a0d4c43703: tablet copy: Starting download of 1 WAL segments...
I20240503 13:56:50.413805 23294 tablet_copy_client.cc:523] T a6a2c05fe3574ab98fb755935b6a92e1 P 61b887f0d40d497badeca5604c1f0546: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:56:50.414227 23299 tablet_copy_client.cc:523] T 3a140864f59b483bb5fea3de83d28ecf P ba0194098aa84449ae17e8a0d4c43703: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:56:50.414984 23294 tablet_bootstrap.cc:492] T a6a2c05fe3574ab98fb755935b6a92e1 P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:50.415473 23299 tablet_bootstrap.cc:492] T 3a140864f59b483bb5fea3de83d28ecf P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:50.419183 23299 tablet_bootstrap.cc:492] T 3a140864f59b483bb5fea3de83d28ecf P ba0194098aa84449ae17e8a0d4c43703: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:56:50.419392 23299 tablet_bootstrap.cc:492] T 3a140864f59b483bb5fea3de83d28ecf P ba0194098aa84449ae17e8a0d4c43703: Bootstrap complete.
I20240503 13:56:50.419533 23299 ts_tablet_manager.cc:1367] T 3a140864f59b483bb5fea3de83d28ecf P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.003s
I20240503 13:56:50.419988 23299 raft_consensus.cc:348] T 3a140864f59b483bb5fea3de83d28ecf P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:50.420121 23299 raft_consensus.cc:729] T 3a140864f59b483bb5fea3de83d28ecf P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:50.420276 23299 consensus_queue.cc:260] T 3a140864f59b483bb5fea3de83d28ecf P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:50.420676 23299 ts_tablet_manager.cc:1398] T 3a140864f59b483bb5fea3de83d28ecf P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 13:56:50.421326 15909 tablet_copy_service.cc:342] P e47105a0f46e4511b7112e571688a4c7: Request end of tablet copy session ba0194098aa84449ae17e8a0d4c43703-3a140864f59b483bb5fea3de83d28ecf received from {username='slave'} at 127.6.98.67:56843
I20240503 13:56:50.421427 15909 tablet_copy_service.cc:433] P e47105a0f46e4511b7112e571688a4c7: ending tablet copy session ba0194098aa84449ae17e8a0d4c43703-3a140864f59b483bb5fea3de83d28ecf on tablet 3a140864f59b483bb5fea3de83d28ecf with peer ba0194098aa84449ae17e8a0d4c43703
I20240503 13:56:50.435388 23294 tablet_bootstrap.cc:492] T a6a2c05fe3574ab98fb755935b6a92e1 P 61b887f0d40d497badeca5604c1f0546: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:56:50.435695 23294 tablet_bootstrap.cc:492] T a6a2c05fe3574ab98fb755935b6a92e1 P 61b887f0d40d497badeca5604c1f0546: Bootstrap complete.
I20240503 13:56:50.435865 23294 ts_tablet_manager.cc:1367] T a6a2c05fe3574ab98fb755935b6a92e1 P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.021s user 0.020s sys 0.000s
I20240503 13:56:50.436270 23294 raft_consensus.cc:348] T a6a2c05fe3574ab98fb755935b6a92e1 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:50.436378 23294 raft_consensus.cc:729] T a6a2c05fe3574ab98fb755935b6a92e1 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:50.436523 23294 consensus_queue.cc:260] T a6a2c05fe3574ab98fb755935b6a92e1 P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:50.436811 23294 ts_tablet_manager.cc:1398] T a6a2c05fe3574ab98fb755935b6a92e1 P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 13:56:50.437299 13205 tablet_copy_service.cc:342] P 8ccf5a4ac450414c9d3579bd5800b770: Request end of tablet copy session 61b887f0d40d497badeca5604c1f0546-a6a2c05fe3574ab98fb755935b6a92e1 received from {username='slave'} at 127.6.98.69:41857
I20240503 13:56:50.437391 13205 tablet_copy_service.cc:433] P 8ccf5a4ac450414c9d3579bd5800b770: ending tablet copy session 61b887f0d40d497badeca5604c1f0546-a6a2c05fe3574ab98fb755935b6a92e1 on tablet a6a2c05fe3574ab98fb755935b6a92e1 with peer 61b887f0d40d497badeca5604c1f0546
I20240503 13:56:50.460893 12910 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:56:50.467998 23292 tablet.cc:1722] T a39982c8458f4291bf292b3436ad8a0a P 7690ee79577c45e3a2c8f8b552c0bff8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:50.468328 23292 tablet.cc:1620] T a39982c8458f4291bf292b3436ad8a0a P 7690ee79577c45e3a2c8f8b552c0bff8: MemRowSet was empty: no flush needed.
I20240503 13:56:50.469524 23293 tablet.cc:1722] T a39982c8458f4291bf292b3436ad8a0a P 8ccf5a4ac450414c9d3579bd5800b770: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:50.469766 23032 tablet.cc:1722] T a39982c8458f4291bf292b3436ad8a0a P e47105a0f46e4511b7112e571688a4c7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:50.469852 23293 tablet.cc:1620] T a39982c8458f4291bf292b3436ad8a0a P 8ccf5a4ac450414c9d3579bd5800b770: MemRowSet was empty: no flush needed.
I20240503 13:56:50.470060 23032 tablet.cc:1620] T a39982c8458f4291bf292b3436ad8a0a P e47105a0f46e4511b7112e571688a4c7: MemRowSet was empty: no flush needed.
I20240503 13:56:50.475862 12910 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=906e52f86b624a34ab9ad0faaf2d8757] alter complete (version 2)
I20240503 13:56:50.480865 22400 raft_consensus.cc:482] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:50.481043 22400 raft_consensus.cc:504] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:50.481496 22400 leader_election.cc:288] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:50.481863 15492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2b9dcff1b72490397f2afc5348923c1" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546" is_pre_election: true
I20240503 13:56:50.481889 15889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2b9dcff1b72490397f2afc5348923c1" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7" is_pre_election: true
I20240503 13:56:50.481994 15492 raft_consensus.cc:2454] T a2b9dcff1b72490397f2afc5348923c1 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 0.
I20240503 13:56:50.482026 15889 raft_consensus.cc:2454] T a2b9dcff1b72490397f2afc5348923c1 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 0.
I20240503 13:56:50.482280 15694 leader_election.cc:302] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61b887f0d40d497badeca5604c1f0546, 76c386bef61a4bd5a907266a5118ffe2; no voters:
I20240503 13:56:50.482442 22400 raft_consensus.cc:2781] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:50.482506 22400 raft_consensus.cc:482] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:50.482550 22400 raft_consensus.cc:3037] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:50.483362 22400 raft_consensus.cc:504] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:50.483681 22400 leader_election.cc:288] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 election: Requested vote from peers 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:50.484006 15492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2b9dcff1b72490397f2afc5348923c1" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546"
I20240503 13:56:50.484045 15889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2b9dcff1b72490397f2afc5348923c1" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7"
I20240503 13:56:50.484112 15492 raft_consensus.cc:3037] T a2b9dcff1b72490397f2afc5348923c1 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:50.484259 15889 raft_consensus.cc:3037] T a2b9dcff1b72490397f2afc5348923c1 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:50.484977 15492 raft_consensus.cc:2454] T a2b9dcff1b72490397f2afc5348923c1 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 1.
I20240503 13:56:50.485255 15694 leader_election.cc:302] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61b887f0d40d497badeca5604c1f0546, 76c386bef61a4bd5a907266a5118ffe2; no voters:
I20240503 13:56:50.485271 15889 raft_consensus.cc:2454] T a2b9dcff1b72490397f2afc5348923c1 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 1.
I20240503 13:56:50.485404 22400 raft_consensus.cc:2781] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:50.485610 22400 raft_consensus.cc:686] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 LEADER]: Becoming Leader. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Running, Role: LEADER
I20240503 13:56:50.485738 22400 consensus_queue.cc:237] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:50.488118 12904 catalog_manager.cc:5484] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71). New cstate: current_term: 1 leader_uuid: "76c386bef61a4bd5a907266a5118ffe2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:50.515981 23325 ts_tablet_manager.cc:902] T 84f1ab11e74045f8a9d7033c5ead0c14 P e47105a0f46e4511b7112e571688a4c7: Initiating tablet copy from peer ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:50.517817 13032 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 13:56:50.518303 15867 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:56:50.519476 23325 tablet_copy_client.cc:308] T 84f1ab11e74045f8a9d7033c5ead0c14 P e47105a0f46e4511b7112e571688a4c7: tablet copy: Beginning tablet copy session from remote peer at address 127.6.98.67:42279
I20240503 13:56:50.520956 13336 tablet_copy_service.cc:140] P ba0194098aa84449ae17e8a0d4c43703: Received BeginTabletCopySession request for tablet 84f1ab11e74045f8a9d7033c5ead0c14 from peer e47105a0f46e4511b7112e571688a4c7 ({username='slave'} at 127.6.98.72:58879)
I20240503 13:56:50.521076 13336 tablet_copy_service.cc:161] P ba0194098aa84449ae17e8a0d4c43703: Beginning new tablet copy session on tablet 84f1ab11e74045f8a9d7033c5ead0c14 from peer e47105a0f46e4511b7112e571688a4c7 at {username='slave'} at 127.6.98.72:58879: session id = e47105a0f46e4511b7112e571688a4c7-84f1ab11e74045f8a9d7033c5ead0c14
I20240503 13:56:50.521111 15472 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 13:56:50.522192 13336 tablet_copy_source_session.cc:205] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:56:50.526695 23325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84f1ab11e74045f8a9d7033c5ead0c14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:50.530571 23325 tablet_copy_client.cc:783] T 84f1ab11e74045f8a9d7033c5ead0c14 P e47105a0f46e4511b7112e571688a4c7: tablet copy: Starting download of 0 data blocks...
I20240503 13:56:50.530767 23325 tablet_copy_client.cc:647] T 84f1ab11e74045f8a9d7033c5ead0c14 P e47105a0f46e4511b7112e571688a4c7: tablet copy: Starting download of 1 WAL segments...
I20240503 13:56:50.531881 23325 tablet_copy_client.cc:523] T 84f1ab11e74045f8a9d7033c5ead0c14 P e47105a0f46e4511b7112e571688a4c7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:56:50.532756 13164 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 13:56:50.535027 23325 tablet_bootstrap.cc:492] T 84f1ab11e74045f8a9d7033c5ead0c14 P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:50.536948 13423 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 13:56:50.537003 13296 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 13:56:50.561385 15604 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:56:50.563752 15736 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 13:56:50.576400 23325 tablet_bootstrap.cc:492] T 84f1ab11e74045f8a9d7033c5ead0c14 P e47105a0f46e4511b7112e571688a4c7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:56:50.576756 23325 tablet_bootstrap.cc:492] T 84f1ab11e74045f8a9d7033c5ead0c14 P e47105a0f46e4511b7112e571688a4c7: Bootstrap complete.
I20240503 13:56:50.576941 23325 ts_tablet_manager.cc:1367] T 84f1ab11e74045f8a9d7033c5ead0c14 P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.042s user 0.027s sys 0.003s
I20240503 13:56:50.577409 23325 raft_consensus.cc:348] T 84f1ab11e74045f8a9d7033c5ead0c14 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:50.577543 23325 raft_consensus.cc:729] T 84f1ab11e74045f8a9d7033c5ead0c14 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:50.577668 23325 consensus_queue.cc:260] T 84f1ab11e74045f8a9d7033c5ead0c14 P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:50.578253 12904 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:56:50.580355 23325 ts_tablet_manager.cc:1398] T 84f1ab11e74045f8a9d7033c5ead0c14 P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:56:50.581564 13336 tablet_copy_service.cc:342] P ba0194098aa84449ae17e8a0d4c43703: Request end of tablet copy session e47105a0f46e4511b7112e571688a4c7-84f1ab11e74045f8a9d7033c5ead0c14 received from {username='slave'} at 127.6.98.72:58879
I20240503 13:56:50.581645 13336 tablet_copy_service.cc:433] P ba0194098aa84449ae17e8a0d4c43703: ending tablet copy session e47105a0f46e4511b7112e571688a4c7-84f1ab11e74045f8a9d7033c5ead0c14 on tablet 84f1ab11e74045f8a9d7033c5ead0c14 with peer e47105a0f46e4511b7112e571688a4c7
I20240503 13:56:50.590109 23292 tablet.cc:1722] T a39982c8458f4291bf292b3436ad8a0a P 7690ee79577c45e3a2c8f8b552c0bff8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:50.595872 12910 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table_new_name [id=906e52f86b624a34ab9ad0faaf2d8757] alter complete (version 3)
I20240503 13:56:50.596071 23032 tablet.cc:1722] T a39982c8458f4291bf292b3436ad8a0a P e47105a0f46e4511b7112e571688a4c7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:50.600903 23293 tablet.cc:1722] T a39982c8458f4291bf292b3436ad8a0a P 8ccf5a4ac450414c9d3579bd5800b770: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:50.608039 20722 consensus_queue.cc:1035] T ada3e4b682c043d585d851c01f689a5c P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:50.609555 23325 ts_tablet_manager.cc:902] T 866723d26350414197ec27544277db9c P e47105a0f46e4511b7112e571688a4c7: Initiating tablet copy from peer ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:50.610016 23325 tablet_copy_client.cc:308] T 866723d26350414197ec27544277db9c P e47105a0f46e4511b7112e571688a4c7: tablet copy: Beginning tablet copy session from remote peer at address 127.6.98.67:42279
I20240503 13:56:50.610481 13336 tablet_copy_service.cc:140] P ba0194098aa84449ae17e8a0d4c43703: Received BeginTabletCopySession request for tablet 866723d26350414197ec27544277db9c from peer e47105a0f46e4511b7112e571688a4c7 ({username='slave'} at 127.6.98.72:58879)
I20240503 13:56:50.610584 13336 tablet_copy_service.cc:161] P ba0194098aa84449ae17e8a0d4c43703: Beginning new tablet copy session on tablet 866723d26350414197ec27544277db9c from peer e47105a0f46e4511b7112e571688a4c7 at {username='slave'} at 127.6.98.72:58879: session id = e47105a0f46e4511b7112e571688a4c7-866723d26350414197ec27544277db9c
I20240503 13:56:50.611274 13336 tablet_copy_source_session.cc:205] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:56:50.612073 23325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 866723d26350414197ec27544277db9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:50.617933 23325 tablet_copy_client.cc:783] T 866723d26350414197ec27544277db9c P e47105a0f46e4511b7112e571688a4c7: tablet copy: Starting download of 0 data blocks...
I20240503 13:56:50.618147 23325 tablet_copy_client.cc:647] T 866723d26350414197ec27544277db9c P e47105a0f46e4511b7112e571688a4c7: tablet copy: Starting download of 1 WAL segments...
I20240503 13:56:50.621767 23325 tablet_copy_client.cc:523] T 866723d26350414197ec27544277db9c P e47105a0f46e4511b7112e571688a4c7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:56:50.623363 23325 tablet_bootstrap.cc:492] T 866723d26350414197ec27544277db9c P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:50.624776 21492 consensus_queue.cc:1035] T ada3e4b682c043d585d851c01f689a5c P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:50.647163 23325 tablet_bootstrap.cc:492] T 866723d26350414197ec27544277db9c P e47105a0f46e4511b7112e571688a4c7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:56:50.647449 23325 tablet_bootstrap.cc:492] T 866723d26350414197ec27544277db9c P e47105a0f46e4511b7112e571688a4c7: Bootstrap complete.
I20240503 13:56:50.647650 23325 ts_tablet_manager.cc:1367] T 866723d26350414197ec27544277db9c P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.024s user 0.018s sys 0.003s
I20240503 13:56:50.648141 23325 raft_consensus.cc:348] T 866723d26350414197ec27544277db9c P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:50.648254 23325 raft_consensus.cc:729] T 866723d26350414197ec27544277db9c P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:50.648375 23325 consensus_queue.cc:260] T 866723d26350414197ec27544277db9c P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:50.648700 23325 ts_tablet_manager.cc:1398] T 866723d26350414197ec27544277db9c P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.001s user 0.000s sys 0.004s
I20240503 13:56:50.649308 13336 tablet_copy_service.cc:342] P ba0194098aa84449ae17e8a0d4c43703: Request end of tablet copy session e47105a0f46e4511b7112e571688a4c7-866723d26350414197ec27544277db9c received from {username='slave'} at 127.6.98.72:58879
I20240503 13:56:50.649395 13336 tablet_copy_service.cc:433] P ba0194098aa84449ae17e8a0d4c43703: ending tablet copy session e47105a0f46e4511b7112e571688a4c7-866723d26350414197ec27544277db9c on tablet 866723d26350414197ec27544277db9c with peer e47105a0f46e4511b7112e571688a4c7
I20240503 13:56:50.697468 12906 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:50.697649 12906 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:50.699851 12906 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Sending DeleteTablet for 3 replicas of tablet a39982c8458f4291bf292b3436ad8a0a
I20240503 13:56:50.700397 15867 tablet_service.cc:1514] Processing DeleteTablet for tablet a39982c8458f4291bf292b3436ad8a0a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:50 UTC) from {username='slave'} at 127.0.0.1:50498
I20240503 13:56:50.700695 13164 tablet_service.cc:1514] Processing DeleteTablet for tablet a39982c8458f4291bf292b3436ad8a0a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:50 UTC) from {username='slave'} at 127.0.0.1:41668
I20240503 13:56:50.700770 23383 tablet_replica.cc:330] T a39982c8458f4291bf292b3436ad8a0a P e47105a0f46e4511b7112e571688a4c7: stopping tablet replica
I20240503 13:56:50.700757 13423 tablet_service.cc:1514] Processing DeleteTablet for tablet a39982c8458f4291bf292b3436ad8a0a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:50 UTC) from {username='slave'} at 127.0.0.1:33678
I20240503 13:56:50.700908 23383 raft_consensus.cc:2229] T a39982c8458f4291bf292b3436ad8a0a P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:50.701069 23383 raft_consensus.cc:2258] T a39982c8458f4291bf292b3436ad8a0a P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:50.701666 23384 tablet_replica.cc:330] T a39982c8458f4291bf292b3436ad8a0a P 8ccf5a4ac450414c9d3579bd5800b770: stopping tablet replica
I20240503 13:56:50.701787 23383 ts_tablet_manager.cc:1875] T a39982c8458f4291bf292b3436ad8a0a P e47105a0f46e4511b7112e571688a4c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:50.701865 23384 raft_consensus.cc:2229] T a39982c8458f4291bf292b3436ad8a0a P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:50.702054 23384 raft_consensus.cc:2258] T a39982c8458f4291bf292b3436ad8a0a P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:50.702651 23385 tablet_replica.cc:330] T a39982c8458f4291bf292b3436ad8a0a P 7690ee79577c45e3a2c8f8b552c0bff8: stopping tablet replica
I20240503 13:56:50.702699 23384 ts_tablet_manager.cc:1875] T a39982c8458f4291bf292b3436ad8a0a P 8ccf5a4ac450414c9d3579bd5800b770: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:50.702967 23385 raft_consensus.cc:2229] T a39982c8458f4291bf292b3436ad8a0a P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:56:50.703231 23385 raft_consensus.cc:2258] T a39982c8458f4291bf292b3436ad8a0a P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:50.703825 23385 ts_tablet_manager.cc:1875] T a39982c8458f4291bf292b3436ad8a0a P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:50.706410 23385 ts_tablet_manager.cc:1888] T a39982c8458f4291bf292b3436ad8a0a P 7690ee79577c45e3a2c8f8b552c0bff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:50.706555 23385 log.cc:1196] T a39982c8458f4291bf292b3436ad8a0a P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal/wals/a39982c8458f4291bf292b3436ad8a0a
I20240503 13:56:50.708135 23385 ts_tablet_manager.cc:1909] T a39982c8458f4291bf292b3436ad8a0a P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting consensus metadata
I20240503 13:56:50.709182 23384 ts_tablet_manager.cc:1888] T a39982c8458f4291bf292b3436ad8a0a P 8ccf5a4ac450414c9d3579bd5800b770: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:50.709336 23384 log.cc:1196] T a39982c8458f4291bf292b3436ad8a0a P 8ccf5a4ac450414c9d3579bd5800b770: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/wal/wals/a39982c8458f4291bf292b3436ad8a0a
I20240503 13:56:50.709654 12896 catalog_manager.cc:4830] TS 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109): tablet a39982c8458f4291bf292b3436ad8a0a (table rebalancer_dynamic_table_new_name [id=906e52f86b624a34ab9ad0faaf2d8757]) successfully deleted
I20240503 13:56:50.709980 23384 ts_tablet_manager.cc:1909] T a39982c8458f4291bf292b3436ad8a0a P 8ccf5a4ac450414c9d3579bd5800b770: Deleting consensus metadata
I20240503 13:56:50.711930 12899 catalog_manager.cc:4830] TS 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387): tablet a39982c8458f4291bf292b3436ad8a0a (table rebalancer_dynamic_table_new_name [id=906e52f86b624a34ab9ad0faaf2d8757]) successfully deleted
I20240503 13:56:50.712714 23383 ts_tablet_manager.cc:1888] T a39982c8458f4291bf292b3436ad8a0a P e47105a0f46e4511b7112e571688a4c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:50.712829 23383 log.cc:1196] T a39982c8458f4291bf292b3436ad8a0a P e47105a0f46e4511b7112e571688a4c7: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-7/wal/wals/a39982c8458f4291bf292b3436ad8a0a
I20240503 13:56:50.713104 23383 ts_tablet_manager.cc:1909] T a39982c8458f4291bf292b3436ad8a0a P e47105a0f46e4511b7112e571688a4c7: Deleting consensus metadata
I20240503 13:56:50.713855 12896 catalog_manager.cc:4830] TS e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): tablet a39982c8458f4291bf292b3436ad8a0a (table rebalancer_dynamic_table_new_name [id=906e52f86b624a34ab9ad0faaf2d8757]) successfully deleted
I20240503 13:56:50.761044 15604 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:56:50.770123 13032 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 13:56:50.780648 13296 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 13:56:50.782016 22738 consensus_queue.cc:1035] T a3418454434b4a9791d50022b8d35743 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:50.788030 13164 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 13:56:50.793540 20870 consensus_queue.cc:1035] T a3418454434b4a9791d50022b8d35743 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:50.804705 15736 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 13:56:50.806808 13423 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:50.806787 12903 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:56:50.810580 15472 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 13:56:50.815368 15604 tablet_service.cc:1467] Processing CreateTablet for tablet eee807daabda4c23a7e5e71d173cffa1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3c8823cb23e845af940d32dedc65b765]), partition=
I20240503 13:56:50.815375 13423 tablet_service.cc:1467] Processing CreateTablet for tablet eee807daabda4c23a7e5e71d173cffa1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3c8823cb23e845af940d32dedc65b765]), partition=
I20240503 13:56:50.815644 13423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eee807daabda4c23a7e5e71d173cffa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:50.815657 15604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eee807daabda4c23a7e5e71d173cffa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:50.818373 13296 tablet_service.cc:1467] Processing CreateTablet for tablet eee807daabda4c23a7e5e71d173cffa1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3c8823cb23e845af940d32dedc65b765]), partition=
I20240503 13:56:50.818605 13296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eee807daabda4c23a7e5e71d173cffa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:50.828800 15867 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:56:50.832618 23404 tablet_bootstrap.cc:492] T eee807daabda4c23a7e5e71d173cffa1 P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:50.833719 23404 tablet_bootstrap.cc:654] T eee807daabda4c23a7e5e71d173cffa1 P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:50.847270 23403 tablet_bootstrap.cc:492] T eee807daabda4c23a7e5e71d173cffa1 P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:50.848922 23403 tablet_bootstrap.cc:654] T eee807daabda4c23a7e5e71d173cffa1 P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:50.851960 23404 tablet_bootstrap.cc:492] T eee807daabda4c23a7e5e71d173cffa1 P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:50.852133 23404 ts_tablet_manager.cc:1367] T eee807daabda4c23a7e5e71d173cffa1 P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20240503 13:56:50.852685 23404 raft_consensus.cc:348] T eee807daabda4c23a7e5e71d173cffa1 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:50.852902 23404 raft_consensus.cc:374] T eee807daabda4c23a7e5e71d173cffa1 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:50.852985 23404 raft_consensus.cc:729] T eee807daabda4c23a7e5e71d173cffa1 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:50.853215 23404 consensus_queue.cc:260] T eee807daabda4c23a7e5e71d173cffa1 P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:50.853601 23404 ts_tablet_manager.cc:1398] T eee807daabda4c23a7e5e71d173cffa1 P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:50.864789 23402 tablet_bootstrap.cc:492] T eee807daabda4c23a7e5e71d173cffa1 P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:50.865940 23402 tablet_bootstrap.cc:654] T eee807daabda4c23a7e5e71d173cffa1 P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:50.868144 23402 tablet_bootstrap.cc:492] T eee807daabda4c23a7e5e71d173cffa1 P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:50.868220 23402 ts_tablet_manager.cc:1367] T eee807daabda4c23a7e5e71d173cffa1 P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:56:50.868626 23402 raft_consensus.cc:348] T eee807daabda4c23a7e5e71d173cffa1 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:50.868718 23402 raft_consensus.cc:374] T eee807daabda4c23a7e5e71d173cffa1 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:50.868736 23402 raft_consensus.cc:729] T eee807daabda4c23a7e5e71d173cffa1 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:50.868835 23402 consensus_queue.cc:260] T eee807daabda4c23a7e5e71d173cffa1 P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:50.869091 23402 ts_tablet_manager.cc:1398] T eee807daabda4c23a7e5e71d173cffa1 P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:50.869856 22400 consensus_queue.cc:1035] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:50.871717 23403 tablet_bootstrap.cc:492] T eee807daabda4c23a7e5e71d173cffa1 P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:50.871814 23403 ts_tablet_manager.cc:1367] T eee807daabda4c23a7e5e71d173cffa1 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.000s
I20240503 13:56:50.872243 23403 raft_consensus.cc:348] T eee807daabda4c23a7e5e71d173cffa1 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:50.872336 23403 raft_consensus.cc:374] T eee807daabda4c23a7e5e71d173cffa1 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:50.872362 23403 raft_consensus.cc:729] T eee807daabda4c23a7e5e71d173cffa1 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:50.872474 23403 consensus_queue.cc:260] T eee807daabda4c23a7e5e71d173cffa1 P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:50.872774 23403 ts_tablet_manager.cc:1398] T eee807daabda4c23a7e5e71d173cffa1 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:50.878721 23131 consensus_queue.cc:1035] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:50.953545 13314 raft_consensus.cc:1203] T 3a140864f59b483bb5fea3de83d28ecf P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 13:56:50.966457 23397 raft_consensus.cc:482] T eee807daabda4c23a7e5e71d173cffa1 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:50.966591 23397 raft_consensus.cc:504] T eee807daabda4c23a7e5e71d173cffa1 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:50.967013 23397 leader_election.cc:288] T eee807daabda4c23a7e5e71d173cffa1 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951), 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:50.967366 15623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eee807daabda4c23a7e5e71d173cffa1" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d" is_pre_election: true
I20240503 13:56:50.967532 15623 raft_consensus.cc:2454] T eee807daabda4c23a7e5e71d173cffa1 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 0.
I20240503 13:56:50.967772 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eee807daabda4c23a7e5e71d173cffa1" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
I20240503 13:56:50.967875 13445 raft_consensus.cc:2454] T eee807daabda4c23a7e5e71d173cffa1 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 0.
I20240503 13:56:50.968271 13249 leader_election.cc:302] T eee807daabda4c23a7e5e71d173cffa1 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:50.968422 23397 raft_consensus.cc:2781] T eee807daabda4c23a7e5e71d173cffa1 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:50.968479 23397 raft_consensus.cc:482] T eee807daabda4c23a7e5e71d173cffa1 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:50.968540 23397 raft_consensus.cc:3037] T eee807daabda4c23a7e5e71d173cffa1 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:50.969614 23397 raft_consensus.cc:504] T eee807daabda4c23a7e5e71d173cffa1 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:50.969780 15493 raft_consensus.cc:1203] T a6a2c05fe3574ab98fb755935b6a92e1 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:56:50.969956 23397 leader_election.cc:288] T eee807daabda4c23a7e5e71d173cffa1 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: Requested vote from peers 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951), 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:50.970430 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eee807daabda4c23a7e5e71d173cffa1" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
I20240503 13:56:50.970546 13445 raft_consensus.cc:3037] T eee807daabda4c23a7e5e71d173cffa1 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:50.971524 13445 raft_consensus.cc:2454] T eee807daabda4c23a7e5e71d173cffa1 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 1.
I20240503 13:56:50.971954 15623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eee807daabda4c23a7e5e71d173cffa1" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d"
I20240503 13:56:50.972082 15623 raft_consensus.cc:3037] T eee807daabda4c23a7e5e71d173cffa1 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:50.973059 15623 raft_consensus.cc:2454] T eee807daabda4c23a7e5e71d173cffa1 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 1.
I20240503 13:56:50.974900 13249 leader_election.cc:302] T eee807daabda4c23a7e5e71d173cffa1 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:50.975443 23397 raft_consensus.cc:2781] T eee807daabda4c23a7e5e71d173cffa1 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:50.975526 23397 raft_consensus.cc:686] T eee807daabda4c23a7e5e71d173cffa1 P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Becoming Leader. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Running, Role: LEADER
I20240503 13:56:50.975653 23397 consensus_queue.cc:237] T eee807daabda4c23a7e5e71d173cffa1 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:50.978122 12911 catalog_manager.cc:5484] T eee807daabda4c23a7e5e71d173cffa1 P ba0194098aa84449ae17e8a0d4c43703 reported cstate change: term changed from 0 to 1, leader changed from <none> to ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67). New cstate: current_term: 1 leader_uuid: "ba0194098aa84449ae17e8a0d4c43703" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } }
I20240503 13:56:51.013280 15604 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:56:51.014758 15472 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 13:56:51.016997 13423 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:51.018218 15736 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 13:56:51.023841 13032 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 13:56:51.032682 13164 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 13:56:51.032696 13296 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:56:51.036916 15867 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:56:51.046521 12912 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_extra_table_56"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:56:51.055269 15888 raft_consensus.cc:1203] T 84f1ab11e74045f8a9d7033c5ead0c14 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:56:51.058519 13032 tablet_service.cc:1467] Processing CreateTablet for tablet 9b11331dcf8a49689c00762da2bfe850 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=c31341be13d94ab88cbc8231d7a4055d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:51.058847 13032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b11331dcf8a49689c00762da2bfe850. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:51.059456 15472 tablet_service.cc:1467] Processing CreateTablet for tablet 1406710f38c2497aad734ae0e41b8808 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=c31341be13d94ab88cbc8231d7a4055d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:51.059718 15472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1406710f38c2497aad734ae0e41b8808. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:51.062548 13296 tablet_service.cc:1467] Processing CreateTablet for tablet 9b11331dcf8a49689c00762da2bfe850 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=c31341be13d94ab88cbc8231d7a4055d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:51.062803 13296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b11331dcf8a49689c00762da2bfe850. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:51.063120 13164 tablet_service.cc:1467] Processing CreateTablet for tablet 1406710f38c2497aad734ae0e41b8808 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=c31341be13d94ab88cbc8231d7a4055d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:51.063275 13423 tablet_service.cc:1467] Processing CreateTablet for tablet 1406710f38c2497aad734ae0e41b8808 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=c31341be13d94ab88cbc8231d7a4055d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:51.063405 13164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1406710f38c2497aad734ae0e41b8808. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:51.063509 13423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1406710f38c2497aad734ae0e41b8808. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:51.064445 13424 tablet_service.cc:1467] Processing CreateTablet for tablet 3fb09a76c1d1463f9b04c47913dcd74a (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=c31341be13d94ab88cbc8231d7a4055d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:51.064738 13424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fb09a76c1d1463f9b04c47913dcd74a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:51.064993 23404 tablet_bootstrap.cc:492] T 9b11331dcf8a49689c00762da2bfe850 P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:51.065953 23404 tablet_bootstrap.cc:654] T 9b11331dcf8a49689c00762da2bfe850 P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:51.066051 13163 tablet_service.cc:1467] Processing CreateTablet for tablet 3fb09a76c1d1463f9b04c47913dcd74a (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=c31341be13d94ab88cbc8231d7a4055d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:51.066283 13163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fb09a76c1d1463f9b04c47913dcd74a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:51.070432 13427 tablet_service.cc:1467] Processing CreateTablet for tablet 9b11331dcf8a49689c00762da2bfe850 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=c31341be13d94ab88cbc8231d7a4055d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:51.070686 13427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b11331dcf8a49689c00762da2bfe850. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:51.070662 15736 tablet_service.cc:1467] Processing CreateTablet for tablet 3fb09a76c1d1463f9b04c47913dcd74a (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=c31341be13d94ab88cbc8231d7a4055d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:51.070914 15736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fb09a76c1d1463f9b04c47913dcd74a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:51.072589 23403 tablet_bootstrap.cc:492] T 1406710f38c2497aad734ae0e41b8808 P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:51.073519 23403 tablet_bootstrap.cc:654] T 1406710f38c2497aad734ae0e41b8808 P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:51.079442 15888 raft_consensus.cc:1203] T 866723d26350414197ec27544277db9c P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:56:51.081409 23464 tablet_bootstrap.cc:492] T 1406710f38c2497aad734ae0e41b8808 P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:51.082479 23464 tablet_bootstrap.cc:654] T 1406710f38c2497aad734ae0e41b8808 P 61b887f0d40d497badeca5604c1f0546: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:51.083250 23468 tablet_bootstrap.cc:492] T 3fb09a76c1d1463f9b04c47913dcd74a P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:51.084342 23468 tablet_bootstrap.cc:654] T 3fb09a76c1d1463f9b04c47913dcd74a P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:51.086099 23468 tablet_bootstrap.cc:492] T 3fb09a76c1d1463f9b04c47913dcd74a P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:51.086287 23468 ts_tablet_manager.cc:1367] T 3fb09a76c1d1463f9b04c47913dcd74a P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:56:51.086865 23468 raft_consensus.cc:348] T 3fb09a76c1d1463f9b04c47913dcd74a P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:51.087091 23468 raft_consensus.cc:374] T 3fb09a76c1d1463f9b04c47913dcd74a P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:51.087174 23468 raft_consensus.cc:729] T 3fb09a76c1d1463f9b04c47913dcd74a P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:51.087508 23468 consensus_queue.cc:260] T 3fb09a76c1d1463f9b04c47913dcd74a P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:51.089051 23403 tablet_bootstrap.cc:492] T 1406710f38c2497aad734ae0e41b8808 P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:51.089159 23403 ts_tablet_manager.cc:1367] T 1406710f38c2497aad734ae0e41b8808 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.002s
I20240503 13:56:51.089212 23468 ts_tablet_manager.cc:1398] T 3fb09a76c1d1463f9b04c47913dcd74a P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:56:51.089612 23403 raft_consensus.cc:348] T 1406710f38c2497aad734ae0e41b8808 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:51.089715 23403 raft_consensus.cc:374] T 1406710f38c2497aad734ae0e41b8808 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:51.089746 23403 raft_consensus.cc:729] T 1406710f38c2497aad734ae0e41b8808 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:51.089859 23403 consensus_queue.cc:260] T 1406710f38c2497aad734ae0e41b8808 P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:51.089857 12912 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:56:51.090181 23403 ts_tablet_manager.cc:1398] T 1406710f38c2497aad734ae0e41b8808 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:51.090342 23403 tablet_bootstrap.cc:492] T 3fb09a76c1d1463f9b04c47913dcd74a P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:51.090561 23464 tablet_bootstrap.cc:492] T 1406710f38c2497aad734ae0e41b8808 P 61b887f0d40d497badeca5604c1f0546: No bootstrap required, opened a new log
I20240503 13:56:51.090652 23464 ts_tablet_manager.cc:1367] T 1406710f38c2497aad734ae0e41b8808 P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 13:56:51.091223 23464 raft_consensus.cc:348] T 1406710f38c2497aad734ae0e41b8808 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:51.091329 23464 raft_consensus.cc:374] T 1406710f38c2497aad734ae0e41b8808 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:51.091424 23403 tablet_bootstrap.cc:654] T 3fb09a76c1d1463f9b04c47913dcd74a P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:51.092294 23464 raft_consensus.cc:729] T 1406710f38c2497aad734ae0e41b8808 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:51.092446 23464 consensus_queue.cc:260] T 1406710f38c2497aad734ae0e41b8808 P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:51.092765 23464 ts_tablet_manager.cc:1398] T 1406710f38c2497aad734ae0e41b8808 P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:51.095134 15623 raft_consensus.cc:1261] T eee807daabda4c23a7e5e71d173cffa1 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:51.095252 13445 raft_consensus.cc:1261] T eee807daabda4c23a7e5e71d173cffa1 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:51.095655 23397 consensus_queue.cc:1035] T eee807daabda4c23a7e5e71d173cffa1 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:51.095816 23002 consensus_queue.cc:1035] T eee807daabda4c23a7e5e71d173cffa1 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:51.096815 23404 tablet_bootstrap.cc:492] T 9b11331dcf8a49689c00762da2bfe850 P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:51.096913 23404 ts_tablet_manager.cc:1367] T 9b11331dcf8a49689c00762da2bfe850 P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.000s
I20240503 13:56:51.097386 23404 raft_consensus.cc:348] T 9b11331dcf8a49689c00762da2bfe850 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:51.097494 23404 raft_consensus.cc:374] T 9b11331dcf8a49689c00762da2bfe850 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:51.097527 23404 raft_consensus.cc:729] T 9b11331dcf8a49689c00762da2bfe850 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:51.097630 23404 consensus_queue.cc:260] T 9b11331dcf8a49689c00762da2bfe850 P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:51.097913 23404 ts_tablet_manager.cc:1398] T 9b11331dcf8a49689c00762da2bfe850 P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:51.099750 12909 catalog_manager.cc:5576] TS 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951) does not have the latest schema for tablet eee807daabda4c23a7e5e71d173cffa1 (table rebalancer_dynamic_table [id=3c8823cb23e845af940d32dedc65b765]). Expected version 1 got 0
I20240503 13:56:51.100667 23467 tablet_bootstrap.cc:492] T 9b11331dcf8a49689c00762da2bfe850 P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:51.101641 23467 tablet_bootstrap.cc:654] T 9b11331dcf8a49689c00762da2bfe850 P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:51.103457 12912 catalog_manager.cc:5576] TS 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951) does not have the latest schema for tablet eee807daabda4c23a7e5e71d173cffa1 (table rebalancer_dynamic_table [id=3c8823cb23e845af940d32dedc65b765]). Expected version 1 got 0
I20240503 13:56:51.104276 23403 tablet_bootstrap.cc:492] T 3fb09a76c1d1463f9b04c47913dcd74a P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:51.104377 23403 ts_tablet_manager.cc:1367] T 3fb09a76c1d1463f9b04c47913dcd74a P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.002s
I20240503 13:56:51.104887 23403 raft_consensus.cc:348] T 3fb09a76c1d1463f9b04c47913dcd74a P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:51.105000 23403 raft_consensus.cc:374] T 3fb09a76c1d1463f9b04c47913dcd74a P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:51.104987 12905 catalog_manager.cc:5576] TS 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109) does not have the latest schema for tablet eee807daabda4c23a7e5e71d173cffa1 (table rebalancer_dynamic_table [id=3c8823cb23e845af940d32dedc65b765]). Expected version 1 got 0
I20240503 13:56:51.105041 23403 raft_consensus.cc:729] T 3fb09a76c1d1463f9b04c47913dcd74a P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:51.105157 23403 consensus_queue.cc:260] T 3fb09a76c1d1463f9b04c47913dcd74a P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:51.105422 23403 ts_tablet_manager.cc:1398] T 3fb09a76c1d1463f9b04c47913dcd74a P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:51.105577 23403 tablet_bootstrap.cc:492] T 9b11331dcf8a49689c00762da2bfe850 P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:51.107688 23403 tablet_bootstrap.cc:654] T 9b11331dcf8a49689c00762da2bfe850 P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:51.110074 23403 tablet_bootstrap.cc:492] T 9b11331dcf8a49689c00762da2bfe850 P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:51.110167 23403 ts_tablet_manager.cc:1367] T 9b11331dcf8a49689c00762da2bfe850 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240503 13:56:51.110618 23403 raft_consensus.cc:348] T 9b11331dcf8a49689c00762da2bfe850 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:51.110721 23403 raft_consensus.cc:374] T 9b11331dcf8a49689c00762da2bfe850 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:51.110745 23403 raft_consensus.cc:729] T 9b11331dcf8a49689c00762da2bfe850 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:51.110834 23403 consensus_queue.cc:260] T 9b11331dcf8a49689c00762da2bfe850 P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:51.111007 23465 tablet_bootstrap.cc:492] T 1406710f38c2497aad734ae0e41b8808 P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:51.111116 23403 ts_tablet_manager.cc:1398] T 9b11331dcf8a49689c00762da2bfe850 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:51.111863 23465 tablet_bootstrap.cc:654] T 1406710f38c2497aad734ae0e41b8808 P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:51.115171 20722 raft_consensus.cc:482] T 1406710f38c2497aad734ae0e41b8808 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:51.115286 20722 raft_consensus.cc:504] T 1406710f38c2497aad734ae0e41b8808 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:51.115798 20722 leader_election.cc:288] T 1406710f38c2497aad734ae0e41b8808 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:51.116068 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1406710f38c2497aad734ae0e41b8808" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
I20240503 13:56:51.116189 13447 raft_consensus.cc:2454] T 1406710f38c2497aad734ae0e41b8808 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 0.
I20240503 13:56:51.116279 13185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1406710f38c2497aad734ae0e41b8808" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770" is_pre_election: true
W20240503 13:56:51.116683 15429 leader_election.cc:341] T 1406710f38c2497aad734ae0e41b8808 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387): Illegal state: must be running to vote when last-logged opid is not known
I20240503 13:56:51.116796 15426 leader_election.cc:302] T 1406710f38c2497aad734ae0e41b8808 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61b887f0d40d497badeca5604c1f0546, 7690ee79577c45e3a2c8f8b552c0bff8; no voters: 8ccf5a4ac450414c9d3579bd5800b770
I20240503 13:56:51.117448 20722 raft_consensus.cc:2781] T 1406710f38c2497aad734ae0e41b8808 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:51.117506 20722 raft_consensus.cc:482] T 1406710f38c2497aad734ae0e41b8808 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:51.117558 20722 raft_consensus.cc:3037] T 1406710f38c2497aad734ae0e41b8808 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:51.118512 23467 tablet_bootstrap.cc:492] T 9b11331dcf8a49689c00762da2bfe850 P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:51.118530 20722 raft_consensus.cc:504] T 1406710f38c2497aad734ae0e41b8808 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:51.118614 23467 ts_tablet_manager.cc:1367] T 9b11331dcf8a49689c00762da2bfe850 P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20240503 13:56:51.118849 20722 leader_election.cc:288] T 1406710f38c2497aad734ae0e41b8808 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 election: Requested vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:51.119122 23465 tablet_bootstrap.cc:492] T 1406710f38c2497aad734ae0e41b8808 P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:51.119233 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1406710f38c2497aad734ae0e41b8808" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
I20240503 13:56:51.119362 13447 raft_consensus.cc:3037] T 1406710f38c2497aad734ae0e41b8808 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:51.119644 23467 raft_consensus.cc:348] T 9b11331dcf8a49689c00762da2bfe850 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:51.119740 23467 raft_consensus.cc:374] T 9b11331dcf8a49689c00762da2bfe850 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:51.119764 23467 raft_consensus.cc:729] T 9b11331dcf8a49689c00762da2bfe850 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:51.119885 23467 consensus_queue.cc:260] T 9b11331dcf8a49689c00762da2bfe850 P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:51.120160 23467 ts_tablet_manager.cc:1398] T 9b11331dcf8a49689c00762da2bfe850 P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:51.120321 23465 ts_tablet_manager.cc:1367] T 1406710f38c2497aad734ae0e41b8808 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 13:56:51.120352 13447 raft_consensus.cc:2454] T 1406710f38c2497aad734ae0e41b8808 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 1.
I20240503 13:56:51.120788 23465 raft_consensus.cc:348] T 1406710f38c2497aad734ae0e41b8808 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:51.120903 23465 raft_consensus.cc:374] T 1406710f38c2497aad734ae0e41b8808 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:51.120985 23465 raft_consensus.cc:729] T 1406710f38c2497aad734ae0e41b8808 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:51.121122 23465 consensus_queue.cc:260] T 1406710f38c2497aad734ae0e41b8808 P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:51.121407 23465 ts_tablet_manager.cc:1398] T 1406710f38c2497aad734ae0e41b8808 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:51.121619 23465 tablet_bootstrap.cc:492] T 3fb09a76c1d1463f9b04c47913dcd74a P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:51.121819 23476 tablet.cc:1722] T eee807daabda4c23a7e5e71d173cffa1 P ba0194098aa84449ae17e8a0d4c43703: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:51.121855 13185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1406710f38c2497aad734ae0e41b8808" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770"
I20240503 13:56:51.122042 13185 raft_consensus.cc:3037] T 1406710f38c2497aad734ae0e41b8808 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:51.122263 15426 leader_election.cc:302] T 1406710f38c2497aad734ae0e41b8808 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61b887f0d40d497badeca5604c1f0546, 7690ee79577c45e3a2c8f8b552c0bff8; no voters:
I20240503 13:56:51.123028 13185 raft_consensus.cc:2454] T 1406710f38c2497aad734ae0e41b8808 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 1.
I20240503 13:56:51.123461 21492 raft_consensus.cc:2781] T 1406710f38c2497aad734ae0e41b8808 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:51.123548 21492 raft_consensus.cc:686] T 1406710f38c2497aad734ae0e41b8808 P 61b887f0d40d497badeca5604c1f0546 [term 1 LEADER]: Becoming Leader. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Running, Role: LEADER
I20240503 13:56:51.123697 21492 consensus_queue.cc:237] T 1406710f38c2497aad734ae0e41b8808 P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:51.124673 23477 tablet.cc:1722] T eee807daabda4c23a7e5e71d173cffa1 P 009f849ee7b54c80a3f5264695047e4d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:51.124950 23477 tablet.cc:1620] T eee807daabda4c23a7e5e71d173cffa1 P 009f849ee7b54c80a3f5264695047e4d: MemRowSet was empty: no flush needed.
I20240503 13:56:51.125247 23476 tablet.cc:1620] T eee807daabda4c23a7e5e71d173cffa1 P ba0194098aa84449ae17e8a0d4c43703: MemRowSet was empty: no flush needed.
I20240503 13:56:51.128046 23465 tablet_bootstrap.cc:654] T 3fb09a76c1d1463f9b04c47913dcd74a P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:51.132004 12909 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=3c8823cb23e845af940d32dedc65b765] alter complete (version 1)
I20240503 13:56:51.132843 23483 tablet.cc:1722] T eee807daabda4c23a7e5e71d173cffa1 P 7690ee79577c45e3a2c8f8b552c0bff8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:51.133172 23465 tablet_bootstrap.cc:492] T 3fb09a76c1d1463f9b04c47913dcd74a P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:51.133271 23465 ts_tablet_manager.cc:1367] T 3fb09a76c1d1463f9b04c47913dcd74a P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 13:56:51.133425 23483 tablet.cc:1620] T eee807daabda4c23a7e5e71d173cffa1 P 7690ee79577c45e3a2c8f8b552c0bff8: MemRowSet was empty: no flush needed.
I20240503 13:56:51.133708 23465 raft_consensus.cc:348] T 3fb09a76c1d1463f9b04c47913dcd74a P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:51.133819 23465 raft_consensus.cc:374] T 3fb09a76c1d1463f9b04c47913dcd74a P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:51.133850 23465 raft_consensus.cc:729] T 3fb09a76c1d1463f9b04c47913dcd74a P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:51.134069 23465 consensus_queue.cc:260] T 3fb09a76c1d1463f9b04c47913dcd74a P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:51.134404 23465 ts_tablet_manager.cc:1398] T 3fb09a76c1d1463f9b04c47913dcd74a P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:51.131978 12912 catalog_manager.cc:5484] T 1406710f38c2497aad734ae0e41b8808 P 61b887f0d40d497badeca5604c1f0546 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61b887f0d40d497badeca5604c1f0546 (127.6.98.69). New cstate: current_term: 1 leader_uuid: "61b887f0d40d497badeca5604c1f0546" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } }
I20240503 13:56:51.137509 23131 raft_consensus.cc:482] T 3fb09a76c1d1463f9b04c47913dcd74a P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:51.137617 23131 raft_consensus.cc:504] T 3fb09a76c1d1463f9b04c47913dcd74a P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:51.137995 23131 leader_election.cc:288] T 3fb09a76c1d1463f9b04c47913dcd74a P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:51.139015 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fb09a76c1d1463f9b04c47913dcd74a" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
I20240503 13:56:51.139181 13447 raft_consensus.cc:2454] T 3fb09a76c1d1463f9b04c47913dcd74a P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 0.
I20240503 13:56:51.139612 15691 leader_election.cc:302] T 3fb09a76c1d1463f9b04c47913dcd74a P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, 76c386bef61a4bd5a907266a5118ffe2; no voters:
I20240503 13:56:51.139796 23131 raft_consensus.cc:2781] T 3fb09a76c1d1463f9b04c47913dcd74a P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:51.139843 23131 raft_consensus.cc:482] T 3fb09a76c1d1463f9b04c47913dcd74a P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:51.139874 23131 raft_consensus.cc:3037] T 3fb09a76c1d1463f9b04c47913dcd74a P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:51.140877 13185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fb09a76c1d1463f9b04c47913dcd74a" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770" is_pre_election: true
I20240503 13:56:51.140901 23131 raft_consensus.cc:504] T 3fb09a76c1d1463f9b04c47913dcd74a P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:51.141007 13185 raft_consensus.cc:2454] T 3fb09a76c1d1463f9b04c47913dcd74a P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 0.
I20240503 13:56:51.141252 23131 leader_election.cc:288] T 3fb09a76c1d1463f9b04c47913dcd74a P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 election: Requested vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:51.141690 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fb09a76c1d1463f9b04c47913dcd74a" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
I20240503 13:56:51.141796 13447 raft_consensus.cc:3037] T 3fb09a76c1d1463f9b04c47913dcd74a P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:51.142691 13447 raft_consensus.cc:2454] T 3fb09a76c1d1463f9b04c47913dcd74a P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 1.
I20240503 13:56:51.142906 13185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fb09a76c1d1463f9b04c47913dcd74a" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770"
I20240503 13:56:51.142961 15691 leader_election.cc:302] T 3fb09a76c1d1463f9b04c47913dcd74a P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, 76c386bef61a4bd5a907266a5118ffe2; no voters:
I20240503 13:56:51.143000 13185 raft_consensus.cc:3037] T 3fb09a76c1d1463f9b04c47913dcd74a P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:51.143127 23131 raft_consensus.cc:2781] T 3fb09a76c1d1463f9b04c47913dcd74a P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:51.143216 23131 raft_consensus.cc:686] T 3fb09a76c1d1463f9b04c47913dcd74a P 76c386bef61a4bd5a907266a5118ffe2 [term 1 LEADER]: Becoming Leader. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Running, Role: LEADER
I20240503 13:56:51.143334 23131 consensus_queue.cc:237] T 3fb09a76c1d1463f9b04c47913dcd74a P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:51.143990 13185 raft_consensus.cc:2454] T 3fb09a76c1d1463f9b04c47913dcd74a P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 1.
I20240503 13:56:51.145874 12912 catalog_manager.cc:5484] T 3fb09a76c1d1463f9b04c47913dcd74a P 76c386bef61a4bd5a907266a5118ffe2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71). New cstate: current_term: 1 leader_uuid: "76c386bef61a4bd5a907266a5118ffe2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } }
I20240503 13:56:51.238548 12906 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:56:51.244390 23476 tablet.cc:1722] T eee807daabda4c23a7e5e71d173cffa1 P ba0194098aa84449ae17e8a0d4c43703: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:51.244783 23476 tablet.cc:1620] T eee807daabda4c23a7e5e71d173cffa1 P ba0194098aa84449ae17e8a0d4c43703: MemRowSet was empty: no flush needed.
I20240503 13:56:51.246629 23477 tablet.cc:1722] T eee807daabda4c23a7e5e71d173cffa1 P 009f849ee7b54c80a3f5264695047e4d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:51.246920 23477 tablet.cc:1620] T eee807daabda4c23a7e5e71d173cffa1 P 009f849ee7b54c80a3f5264695047e4d: MemRowSet was empty: no flush needed.
I20240503 13:56:51.247349 23483 tablet.cc:1722] T eee807daabda4c23a7e5e71d173cffa1 P 7690ee79577c45e3a2c8f8b552c0bff8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:51.247642 23483 tablet.cc:1620] T eee807daabda4c23a7e5e71d173cffa1 P 7690ee79577c45e3a2c8f8b552c0bff8: MemRowSet was empty: no flush needed.
I20240503 13:56:51.253952 12905 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=3c8823cb23e845af940d32dedc65b765] alter complete (version 2)
I20240503 13:56:51.301440 22738 raft_consensus.cc:482] T 9b11331dcf8a49689c00762da2bfe850 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:51.301576 22738 raft_consensus.cc:504] T 9b11331dcf8a49689c00762da2bfe850 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:51.301956 22738 leader_election.cc:288] T 9b11331dcf8a49689c00762da2bfe850 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:51.302402 13314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b11331dcf8a49689c00762da2bfe850" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703" is_pre_election: true
I20240503 13:56:51.302497 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b11331dcf8a49689c00762da2bfe850" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
I20240503 13:56:51.302538 13314 raft_consensus.cc:2454] T 9b11331dcf8a49689c00762da2bfe850 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 0.
I20240503 13:56:51.302637 13447 raft_consensus.cc:2454] T 9b11331dcf8a49689c00762da2bfe850 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 0.
I20240503 13:56:51.302845 12990 leader_election.cc:302] T 9b11331dcf8a49689c00762da2bfe850 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:51.303033 22738 raft_consensus.cc:2781] T 9b11331dcf8a49689c00762da2bfe850 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:51.303092 22738 raft_consensus.cc:482] T 9b11331dcf8a49689c00762da2bfe850 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:51.303122 22738 raft_consensus.cc:3037] T 9b11331dcf8a49689c00762da2bfe850 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:51.303979 22738 raft_consensus.cc:504] T 9b11331dcf8a49689c00762da2bfe850 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:51.304294 22738 leader_election.cc:288] T 9b11331dcf8a49689c00762da2bfe850 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 election: Requested vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:51.304714 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b11331dcf8a49689c00762da2bfe850" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
I20240503 13:56:51.304728 13314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b11331dcf8a49689c00762da2bfe850" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703"
I20240503 13:56:51.304836 13314 raft_consensus.cc:3037] T 9b11331dcf8a49689c00762da2bfe850 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:51.304837 13447 raft_consensus.cc:3037] T 9b11331dcf8a49689c00762da2bfe850 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:51.305745 13447 raft_consensus.cc:2454] T 9b11331dcf8a49689c00762da2bfe850 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 1.
I20240503 13:56:51.305759 13314 raft_consensus.cc:2454] T 9b11331dcf8a49689c00762da2bfe850 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 1.
I20240503 13:56:51.306085 12987 leader_election.cc:302] T 9b11331dcf8a49689c00762da2bfe850 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 7690ee79577c45e3a2c8f8b552c0bff8; no voters:
I20240503 13:56:51.306293 22738 raft_consensus.cc:2781] T 9b11331dcf8a49689c00762da2bfe850 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:51.306383 22738 raft_consensus.cc:686] T 9b11331dcf8a49689c00762da2bfe850 P 60ba3ffe3b7746469834cd91c84be14e [term 1 LEADER]: Becoming Leader. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Running, Role: LEADER
I20240503 13:56:51.306517 22738 consensus_queue.cc:237] T 9b11331dcf8a49689c00762da2bfe850 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:51.308986 12911 catalog_manager.cc:5484] T 9b11331dcf8a49689c00762da2bfe850 P 60ba3ffe3b7746469834cd91c84be14e reported cstate change: term changed from 0 to 1, leader changed from <none> to 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65). New cstate: current_term: 1 leader_uuid: "60ba3ffe3b7746469834cd91c84be14e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:51.336432 13032 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 13:56:51.340922 13427 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:51.344741 15472 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 13:56:51.345059 15736 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 13:56:51.348263 15604 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:56:51.351851 15867 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:56:51.362893 13163 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 13:56:51.365980 13296 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:56:51.366493 12911 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:56:51.380903 23476 tablet.cc:1722] T eee807daabda4c23a7e5e71d173cffa1 P ba0194098aa84449ae17e8a0d4c43703: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:51.384033 23483 tablet.cc:1722] T eee807daabda4c23a7e5e71d173cffa1 P 7690ee79577c45e3a2c8f8b552c0bff8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:51.389997 12904 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table_new_name [id=3c8823cb23e845af940d32dedc65b765] alter complete (version 3)
I20240503 13:56:51.396737 23477 tablet.cc:1722] T eee807daabda4c23a7e5e71d173cffa1 P 009f849ee7b54c80a3f5264695047e4d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:51.451174 13185 consensus_queue.cc:237] T a6a2c05fe3574ab98fb755935b6a92e1 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:51.452558 13051 raft_consensus.cc:1261] T a6a2c05fe3574ab98fb755935b6a92e1 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Refusing update from remote peer 8ccf5a4ac450414c9d3579bd5800b770: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:56:51.452755 15491 raft_consensus.cc:1261] T a6a2c05fe3574ab98fb755935b6a92e1 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Refusing update from remote peer 8ccf5a4ac450414c9d3579bd5800b770: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:56:51.452950 23306 consensus_queue.cc:1035] T a6a2c05fe3574ab98fb755935b6a92e1 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:51.453140 23143 consensus_queue.cc:1035] T a6a2c05fe3574ab98fb755935b6a92e1 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:51.455179 23143 raft_consensus.cc:2932] T a6a2c05fe3574ab98fb755935b6a92e1 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:51.455616 13051 raft_consensus.cc:2932] T a6a2c05fe3574ab98fb755935b6a92e1 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:51.455978 15491 raft_consensus.cc:2932] T a6a2c05fe3574ab98fb755935b6a92e1 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:51.458570 12911 catalog_manager.cc:5484] T a6a2c05fe3574ab98fb755935b6a92e1 P 8ccf5a4ac450414c9d3579bd5800b770 reported cstate change: config changed from index 4 to 5, VOTER ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) evicted. New cstate: current_term: 1 leader_uuid: "8ccf5a4ac450414c9d3579bd5800b770" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:51.461673 13296 tablet_service.cc:1514] Processing DeleteTablet for tablet a6a2c05fe3574ab98fb755935b6a92e1 with delete_type TABLET_DATA_TOMBSTONED (TS ba0194098aa84449ae17e8a0d4c43703 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:33250
I20240503 13:56:51.462055 23556 tablet_replica.cc:330] T a6a2c05fe3574ab98fb755935b6a92e1 P ba0194098aa84449ae17e8a0d4c43703: stopping tablet replica
I20240503 13:56:51.462383 23556 raft_consensus.cc:2229] T a6a2c05fe3574ab98fb755935b6a92e1 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:51.462718 23556 raft_consensus.cc:2258] T a6a2c05fe3574ab98fb755935b6a92e1 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:51.463271 23556 ts_tablet_manager.cc:1875] T a6a2c05fe3574ab98fb755935b6a92e1 P ba0194098aa84449ae17e8a0d4c43703: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:56:51.465585 23556 ts_tablet_manager.cc:1888] T a6a2c05fe3574ab98fb755935b6a92e1 P ba0194098aa84449ae17e8a0d4c43703: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 13:56:51.465662 23556 log.cc:1196] T a6a2c05fe3574ab98fb755935b6a92e1 P ba0194098aa84449ae17e8a0d4c43703: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/wal/wals/a6a2c05fe3574ab98fb755935b6a92e1
I20240503 13:56:51.466197 12899 catalog_manager.cc:4830] TS ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279): tablet a6a2c05fe3574ab98fb755935b6a92e1 (table rebalance_test_table_13 [id=4d62f84fd7bc47e8a1f2de384d64b787]) successfully deleted
I20240503 13:56:51.495335 12904 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:51.495486 12904 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:51.497758 12904 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Sending DeleteTablet for 3 replicas of tablet eee807daabda4c23a7e5e71d173cffa1
I20240503 13:56:51.498505 13296 tablet_service.cc:1514] Processing DeleteTablet for tablet eee807daabda4c23a7e5e71d173cffa1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:51 UTC) from {username='slave'} at 127.0.0.1:33250
I20240503 13:56:51.498690 23556 tablet_replica.cc:330] T eee807daabda4c23a7e5e71d173cffa1 P ba0194098aa84449ae17e8a0d4c43703: stopping tablet replica
I20240503 13:56:51.498839 23556 raft_consensus.cc:2229] T eee807daabda4c23a7e5e71d173cffa1 P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:56:51.498837 13427 tablet_service.cc:1514] Processing DeleteTablet for tablet eee807daabda4c23a7e5e71d173cffa1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:51 UTC) from {username='slave'} at 127.0.0.1:33678
I20240503 13:56:51.499045 23556 raft_consensus.cc:2258] T eee807daabda4c23a7e5e71d173cffa1 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:51.499127 15604 tablet_service.cc:1514] Processing DeleteTablet for tablet eee807daabda4c23a7e5e71d173cffa1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:51 UTC) from {username='slave'} at 127.0.0.1:60718
I20240503 13:56:51.499446 23579 tablet_replica.cc:330] T eee807daabda4c23a7e5e71d173cffa1 P 009f849ee7b54c80a3f5264695047e4d: stopping tablet replica
I20240503 13:56:51.499469 23578 tablet_replica.cc:330] T eee807daabda4c23a7e5e71d173cffa1 P 7690ee79577c45e3a2c8f8b552c0bff8: stopping tablet replica
I20240503 13:56:51.499663 23579 raft_consensus.cc:2229] T eee807daabda4c23a7e5e71d173cffa1 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:51.499840 23556 ts_tablet_manager.cc:1875] T eee807daabda4c23a7e5e71d173cffa1 P ba0194098aa84449ae17e8a0d4c43703: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:51.499877 23579 raft_consensus.cc:2258] T eee807daabda4c23a7e5e71d173cffa1 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:51.500226 23578 raft_consensus.cc:2229] T eee807daabda4c23a7e5e71d173cffa1 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:51.500361 23578 raft_consensus.cc:2258] T eee807daabda4c23a7e5e71d173cffa1 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:51.501034 23579 ts_tablet_manager.cc:1875] T eee807daabda4c23a7e5e71d173cffa1 P 009f849ee7b54c80a3f5264695047e4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:51.501114 23578 ts_tablet_manager.cc:1875] T eee807daabda4c23a7e5e71d173cffa1 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:51.502242 23556 ts_tablet_manager.cc:1888] T eee807daabda4c23a7e5e71d173cffa1 P ba0194098aa84449ae17e8a0d4c43703: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:51.502305 23556 log.cc:1196] T eee807daabda4c23a7e5e71d173cffa1 P ba0194098aa84449ae17e8a0d4c43703: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/wal/wals/eee807daabda4c23a7e5e71d173cffa1
I20240503 13:56:51.502557 23556 ts_tablet_manager.cc:1909] T eee807daabda4c23a7e5e71d173cffa1 P ba0194098aa84449ae17e8a0d4c43703: Deleting consensus metadata
I20240503 13:56:51.503536 23579 ts_tablet_manager.cc:1888] T eee807daabda4c23a7e5e71d173cffa1 P 009f849ee7b54c80a3f5264695047e4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:51.503607 23579 log.cc:1196] T eee807daabda4c23a7e5e71d173cffa1 P 009f849ee7b54c80a3f5264695047e4d: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/wal/wals/eee807daabda4c23a7e5e71d173cffa1
I20240503 13:56:51.503850 23579 ts_tablet_manager.cc:1909] T eee807daabda4c23a7e5e71d173cffa1 P 009f849ee7b54c80a3f5264695047e4d: Deleting consensus metadata
I20240503 13:56:51.504035 23578 ts_tablet_manager.cc:1888] T eee807daabda4c23a7e5e71d173cffa1 P 7690ee79577c45e3a2c8f8b552c0bff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:51.504161 23578 log.cc:1196] T eee807daabda4c23a7e5e71d173cffa1 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal/wals/eee807daabda4c23a7e5e71d173cffa1
I20240503 13:56:51.504626 12896 catalog_manager.cc:4830] TS 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951): tablet eee807daabda4c23a7e5e71d173cffa1 (table rebalancer_dynamic_table_new_name [id=3c8823cb23e845af940d32dedc65b765]) successfully deleted
I20240503 13:56:51.504940 23578 ts_tablet_manager.cc:1909] T eee807daabda4c23a7e5e71d173cffa1 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting consensus metadata
I20240503 13:56:51.505584 12896 catalog_manager.cc:4830] TS 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109): tablet eee807daabda4c23a7e5e71d173cffa1 (table rebalancer_dynamic_table_new_name [id=3c8823cb23e845af940d32dedc65b765]) successfully deleted
I20240503 13:56:51.505842 12899 catalog_manager.cc:4830] TS ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279): tablet eee807daabda4c23a7e5e71d173cffa1 (table rebalancer_dynamic_table_new_name [id=3c8823cb23e845af940d32dedc65b765]) successfully deleted
I20240503 13:56:51.583305 15604 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:56:51.584220 13032 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 13:56:51.588040 22400 consensus_queue.cc:1035] T 3fb09a76c1d1463f9b04c47913dcd74a P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:51.595032 15472 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 13:56:51.601651 13427 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:51.602957 12912 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:56:51.605417 15736 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 13:56:51.611644 13296 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 13:56:51.615592 15736 tablet_service.cc:1467] Processing CreateTablet for tablet 7f98c13e21054e6ba8a52827f91c60e8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=75c555d294b9416b8478d1b0898b2cba]), partition=
I20240503 13:56:51.615845 15736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f98c13e21054e6ba8a52827f91c60e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:51.621296 15867 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:56:51.621606 15604 tablet_service.cc:1467] Processing CreateTablet for tablet 7f98c13e21054e6ba8a52827f91c60e8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=75c555d294b9416b8478d1b0898b2cba]), partition=
I20240503 13:56:51.621873 15604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f98c13e21054e6ba8a52827f91c60e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:51.633726 15472 tablet_service.cc:1467] Processing CreateTablet for tablet 7f98c13e21054e6ba8a52827f91c60e8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=75c555d294b9416b8478d1b0898b2cba]), partition=
I20240503 13:56:51.633999 15472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f98c13e21054e6ba8a52827f91c60e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:51.637077 23616 tablet_bootstrap.cc:492] T 7f98c13e21054e6ba8a52827f91c60e8 P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:51.638135 23616 tablet_bootstrap.cc:654] T 7f98c13e21054e6ba8a52827f91c60e8 P 61b887f0d40d497badeca5604c1f0546: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:51.638463 23094 consensus_queue.cc:1035] T 1406710f38c2497aad734ae0e41b8808 P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:51.639899 23131 consensus_queue.cc:1035] T 3fb09a76c1d1463f9b04c47913dcd74a P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:51.663961 13163 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 13:56:51.675258 23612 tablet_bootstrap.cc:492] T 7f98c13e21054e6ba8a52827f91c60e8 P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:51.676371 23612 tablet_bootstrap.cc:654] T 7f98c13e21054e6ba8a52827f91c60e8 P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:51.677095 23613 tablet_bootstrap.cc:492] T 7f98c13e21054e6ba8a52827f91c60e8 P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:51.677825 23613 tablet_bootstrap.cc:654] T 7f98c13e21054e6ba8a52827f91c60e8 P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:51.678632 23612 tablet_bootstrap.cc:492] T 7f98c13e21054e6ba8a52827f91c60e8 P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:51.678731 23612 ts_tablet_manager.cc:1367] T 7f98c13e21054e6ba8a52827f91c60e8 P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:56:51.678869 23613 tablet_bootstrap.cc:492] T 7f98c13e21054e6ba8a52827f91c60e8 P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:51.678951 23613 ts_tablet_manager.cc:1367] T 7f98c13e21054e6ba8a52827f91c60e8 P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:51.679208 23612 raft_consensus.cc:348] T 7f98c13e21054e6ba8a52827f91c60e8 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:51.679455 23612 raft_consensus.cc:374] T 7f98c13e21054e6ba8a52827f91c60e8 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:51.679570 23612 raft_consensus.cc:729] T 7f98c13e21054e6ba8a52827f91c60e8 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:51.679610 23613 raft_consensus.cc:348] T 7f98c13e21054e6ba8a52827f91c60e8 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:51.679737 23613 raft_consensus.cc:374] T 7f98c13e21054e6ba8a52827f91c60e8 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:51.679793 23613 raft_consensus.cc:729] T 7f98c13e21054e6ba8a52827f91c60e8 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:51.679816 23612 consensus_queue.cc:260] T 7f98c13e21054e6ba8a52827f91c60e8 P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:51.679955 23613 consensus_queue.cc:260] T 7f98c13e21054e6ba8a52827f91c60e8 P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:51.680289 23612 ts_tablet_manager.cc:1398] T 7f98c13e21054e6ba8a52827f91c60e8 P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:51.681258 23613 ts_tablet_manager.cc:1398] T 7f98c13e21054e6ba8a52827f91c60e8 P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:51.682024 23616 tablet_bootstrap.cc:492] T 7f98c13e21054e6ba8a52827f91c60e8 P 61b887f0d40d497badeca5604c1f0546: No bootstrap required, opened a new log
I20240503 13:56:51.682140 23616 ts_tablet_manager.cc:1367] T 7f98c13e21054e6ba8a52827f91c60e8 P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.045s user 0.002s sys 0.000s
I20240503 13:56:51.682623 23616 raft_consensus.cc:348] T 7f98c13e21054e6ba8a52827f91c60e8 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:51.682791 23616 raft_consensus.cc:374] T 7f98c13e21054e6ba8a52827f91c60e8 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:51.682862 23616 raft_consensus.cc:729] T 7f98c13e21054e6ba8a52827f91c60e8 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:51.683161 23616 consensus_queue.cc:260] T 7f98c13e21054e6ba8a52827f91c60e8 P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:51.683625 23616 ts_tablet_manager.cc:1398] T 7f98c13e21054e6ba8a52827f91c60e8 P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:51.688267 23094 consensus_queue.cc:1035] T 1406710f38c2497aad734ae0e41b8808 P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:51.724889 15492 consensus_queue.cc:237] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:51.728344 15624 raft_consensus.cc:1261] T c7d5e8c09ec14ac7891c8a63cb896878 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Refusing update from remote peer 61b887f0d40d497badeca5604c1f0546: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:56:51.728521 15889 raft_consensus.cc:1261] T c7d5e8c09ec14ac7891c8a63cb896878 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Refusing update from remote peer 61b887f0d40d497badeca5604c1f0546: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:56:51.730285 23094 consensus_queue.cc:1035] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:56:51.730568 23642 consensus_queue.cc:1035] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:56:51.732183 23094 raft_consensus.cc:2932] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:51.733131 15889 raft_consensus.cc:2932] T c7d5e8c09ec14ac7891c8a63cb896878 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:51.734526 15624 raft_consensus.cc:2932] T c7d5e8c09ec14ac7891c8a63cb896878 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:51.735424 12912 catalog_manager.cc:5484] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 reported cstate change: config changed from index 2 to 3, VOTER 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68) evicted. New cstate: current_term: 1 leader_uuid: "61b887f0d40d497badeca5604c1f0546" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:51.738282 13427 tablet_service.cc:1514] Processing DeleteTablet for tablet c7d5e8c09ec14ac7891c8a63cb896878 with delete_type TABLET_DATA_TOMBSTONED (TS 7690ee79577c45e3a2c8f8b552c0bff8 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:33678
I20240503 13:56:51.738548 23578 tablet_replica.cc:330] T c7d5e8c09ec14ac7891c8a63cb896878 P 7690ee79577c45e3a2c8f8b552c0bff8: stopping tablet replica
I20240503 13:56:51.738687 23578 raft_consensus.cc:2229] T c7d5e8c09ec14ac7891c8a63cb896878 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:51.738811 23578 raft_consensus.cc:2258] T c7d5e8c09ec14ac7891c8a63cb896878 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:51.739248 23578 ts_tablet_manager.cc:1875] T c7d5e8c09ec14ac7891c8a63cb896878 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:56:51.741478 23578 ts_tablet_manager.cc:1888] T c7d5e8c09ec14ac7891c8a63cb896878 P 7690ee79577c45e3a2c8f8b552c0bff8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 13:56:51.741554 23578 log.cc:1196] T c7d5e8c09ec14ac7891c8a63cb896878 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal/wals/c7d5e8c09ec14ac7891c8a63cb896878
I20240503 13:56:51.742060 12896 catalog_manager.cc:4830] TS 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109): tablet c7d5e8c09ec14ac7891c8a63cb896878 (table rebalancer_extra_table_0 [id=18c0e326c0a144c382cd95a8c0052057]) successfully deleted
I20240503 13:56:51.839119 22738 consensus_queue.cc:1035] T 9b11331dcf8a49689c00762da2bfe850 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:51.840350 23642 raft_consensus.cc:482] T 7f98c13e21054e6ba8a52827f91c60e8 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:51.840477 23642 raft_consensus.cc:504] T 7f98c13e21054e6ba8a52827f91c60e8 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:51.840987 23642 leader_election.cc:288] T 7f98c13e21054e6ba8a52827f91c60e8 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951), 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885)
I20240503 13:56:51.841423 15624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f98c13e21054e6ba8a52827f91c60e8" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d" is_pre_election: true
I20240503 13:56:51.841560 15624 raft_consensus.cc:2454] T 7f98c13e21054e6ba8a52827f91c60e8 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 0.
I20240503 13:56:51.841885 15426 leader_election.cc:302] T 7f98c13e21054e6ba8a52827f91c60e8 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 61b887f0d40d497badeca5604c1f0546; no voters:
I20240503 13:56:51.841961 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f98c13e21054e6ba8a52827f91c60e8" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2" is_pre_election: true
I20240503 13:56:51.842100 15757 raft_consensus.cc:2454] T 7f98c13e21054e6ba8a52827f91c60e8 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 0.
I20240503 13:56:51.842438 23642 raft_consensus.cc:2781] T 7f98c13e21054e6ba8a52827f91c60e8 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:51.842497 23642 raft_consensus.cc:482] T 7f98c13e21054e6ba8a52827f91c60e8 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:51.842535 23642 raft_consensus.cc:3037] T 7f98c13e21054e6ba8a52827f91c60e8 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:51.843662 23642 raft_consensus.cc:504] T 7f98c13e21054e6ba8a52827f91c60e8 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:51.844031 23642 leader_election.cc:288] T 7f98c13e21054e6ba8a52827f91c60e8 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 election: Requested vote from peers 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951), 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885)
I20240503 13:56:51.846128 15624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f98c13e21054e6ba8a52827f91c60e8" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d"
I20240503 13:56:51.846247 15624 raft_consensus.cc:3037] T 7f98c13e21054e6ba8a52827f91c60e8 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:51.847280 15624 raft_consensus.cc:2454] T 7f98c13e21054e6ba8a52827f91c60e8 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 1.
I20240503 13:56:51.849483 15758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f98c13e21054e6ba8a52827f91c60e8" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2"
I20240503 13:56:51.849614 15758 raft_consensus.cc:3037] T 7f98c13e21054e6ba8a52827f91c60e8 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:51.850699 15758 raft_consensus.cc:2454] T 7f98c13e21054e6ba8a52827f91c60e8 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 1.
I20240503 13:56:51.851245 15426 leader_election.cc:302] T 7f98c13e21054e6ba8a52827f91c60e8 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 61b887f0d40d497badeca5604c1f0546; no voters:
I20240503 13:56:51.851511 20870 consensus_queue.cc:1035] T 9b11331dcf8a49689c00762da2bfe850 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:51.852100 12910 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_extra_table_58"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:56:51.857868 23642 raft_consensus.cc:2781] T 7f98c13e21054e6ba8a52827f91c60e8 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:51.858006 23642 raft_consensus.cc:686] T 7f98c13e21054e6ba8a52827f91c60e8 P 61b887f0d40d497badeca5604c1f0546 [term 1 LEADER]: Becoming Leader. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Running, Role: LEADER
I20240503 13:56:51.858139 23642 consensus_queue.cc:237] T 7f98c13e21054e6ba8a52827f91c60e8 P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:51.859913 15867 tablet_service.cc:1467] Processing CreateTablet for tablet aa97788f7cdb4caaa243a609b34f55cf (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=62c61e6a8f154573861f621e3546f7cf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:51.860241 15867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa97788f7cdb4caaa243a609b34f55cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:51.860405 15604 tablet_service.cc:1467] Processing CreateTablet for tablet bc91cdcb85a24e869a250a882927abd8 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=62c61e6a8f154573861f621e3546f7cf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:51.860721 15604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc91cdcb85a24e869a250a882927abd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:51.860634 12910 catalog_manager.cc:5484] T 7f98c13e21054e6ba8a52827f91c60e8 P 61b887f0d40d497badeca5604c1f0546 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61b887f0d40d497badeca5604c1f0546 (127.6.98.69). New cstate: current_term: 1 leader_uuid: "61b887f0d40d497badeca5604c1f0546" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:51.860769 13427 tablet_service.cc:1467] Processing CreateTablet for tablet bc91cdcb85a24e869a250a882927abd8 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=62c61e6a8f154573861f621e3546f7cf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:51.861048 13427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc91cdcb85a24e869a250a882927abd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:51.861212 13296 tablet_service.cc:1467] Processing CreateTablet for tablet aa97788f7cdb4caaa243a609b34f55cf (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=62c61e6a8f154573861f621e3546f7cf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:51.861459 13296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa97788f7cdb4caaa243a609b34f55cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:51.861513 13163 tablet_service.cc:1467] Processing CreateTablet for tablet aa97788f7cdb4caaa243a609b34f55cf (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=62c61e6a8f154573861f621e3546f7cf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:51.861773 13163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa97788f7cdb4caaa243a609b34f55cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:51.863126 13295 tablet_service.cc:1467] Processing CreateTablet for tablet bc91cdcb85a24e869a250a882927abd8 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=62c61e6a8f154573861f621e3546f7cf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:51.863341 13295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc91cdcb85a24e869a250a882927abd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:51.865666 15736 tablet_service.cc:1467] Processing CreateTablet for tablet 0cda7f68204345cfa501efbb822135c2 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=62c61e6a8f154573861f621e3546f7cf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:51.865952 15736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cda7f68204345cfa501efbb822135c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:51.867449 23692 tablet_bootstrap.cc:492] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:51.867549 23691 tablet_bootstrap.cc:492] T bc91cdcb85a24e869a250a882927abd8 P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:51.868358 23613 tablet_bootstrap.cc:492] T bc91cdcb85a24e869a250a882927abd8 P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:51.868387 23694 tablet_bootstrap.cc:492] T aa97788f7cdb4caaa243a609b34f55cf P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:51.868856 23691 tablet_bootstrap.cc:654] T bc91cdcb85a24e869a250a882927abd8 P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:51.869133 23694 tablet_bootstrap.cc:654] T aa97788f7cdb4caaa243a609b34f55cf P e47105a0f46e4511b7112e571688a4c7: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:51.869279 23613 tablet_bootstrap.cc:654] T bc91cdcb85a24e869a250a882927abd8 P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:51.869567 23693 tablet_bootstrap.cc:492] T aa97788f7cdb4caaa243a609b34f55cf P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:51.869709 23692 tablet_bootstrap.cc:654] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:51.870200 23612 tablet_bootstrap.cc:492] T 0cda7f68204345cfa501efbb822135c2 P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:51.870478 23693 tablet_bootstrap.cc:654] T aa97788f7cdb4caaa243a609b34f55cf P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:51.870653 23692 tablet_bootstrap.cc:492] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:51.870714 23692 ts_tablet_manager.cc:1367] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:56:51.871109 23692 raft_consensus.cc:348] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:51.871212 23692 raft_consensus.cc:374] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:51.871240 23692 raft_consensus.cc:729] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:51.871265 23612 tablet_bootstrap.cc:654] T 0cda7f68204345cfa501efbb822135c2 P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:51.871351 23692 consensus_queue.cc:260] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:51.871600 23692 ts_tablet_manager.cc:1398] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:51.871711 23692 tablet_bootstrap.cc:492] T bc91cdcb85a24e869a250a882927abd8 P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:51.872509 23692 tablet_bootstrap.cc:654] T bc91cdcb85a24e869a250a882927abd8 P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:51.873169 23691 tablet_bootstrap.cc:492] T bc91cdcb85a24e869a250a882927abd8 P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:51.873263 23691 ts_tablet_manager.cc:1367] T bc91cdcb85a24e869a250a882927abd8 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 13:56:51.873715 23691 raft_consensus.cc:348] T bc91cdcb85a24e869a250a882927abd8 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:51.873823 23691 raft_consensus.cc:374] T bc91cdcb85a24e869a250a882927abd8 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:51.873857 23691 raft_consensus.cc:729] T bc91cdcb85a24e869a250a882927abd8 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:51.873979 23692 tablet_bootstrap.cc:492] T bc91cdcb85a24e869a250a882927abd8 P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:51.873980 23691 consensus_queue.cc:260] T bc91cdcb85a24e869a250a882927abd8 P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:51.874047 23692 ts_tablet_manager.cc:1367] T bc91cdcb85a24e869a250a882927abd8 P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:56:51.874258 23691 ts_tablet_manager.cc:1398] T bc91cdcb85a24e869a250a882927abd8 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:51.874850 13427 tablet_service.cc:1467] Processing CreateTablet for tablet 0cda7f68204345cfa501efbb822135c2 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=62c61e6a8f154573861f621e3546f7cf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:51.875113 13427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cda7f68204345cfa501efbb822135c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:51.875326 23694 tablet_bootstrap.cc:492] T aa97788f7cdb4caaa243a609b34f55cf P e47105a0f46e4511b7112e571688a4c7: No bootstrap required, opened a new log
I20240503 13:56:51.875411 23694 ts_tablet_manager.cc:1367] T aa97788f7cdb4caaa243a609b34f55cf P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240503 13:56:51.875857 23694 raft_consensus.cc:348] T aa97788f7cdb4caaa243a609b34f55cf P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:51.875967 23694 raft_consensus.cc:374] T aa97788f7cdb4caaa243a609b34f55cf P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:51.875998 23694 raft_consensus.cc:729] T aa97788f7cdb4caaa243a609b34f55cf P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:51.876111 23694 consensus_queue.cc:260] T aa97788f7cdb4caaa243a609b34f55cf P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:51.876389 23694 ts_tablet_manager.cc:1398] T aa97788f7cdb4caaa243a609b34f55cf P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:51.877871 23691 tablet_bootstrap.cc:492] T 0cda7f68204345cfa501efbb822135c2 P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:51.878135 23613 tablet_bootstrap.cc:492] T bc91cdcb85a24e869a250a882927abd8 P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:51.878232 23613 ts_tablet_manager.cc:1367] T bc91cdcb85a24e869a250a882927abd8 P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 13:56:51.878720 23613 raft_consensus.cc:348] T bc91cdcb85a24e869a250a882927abd8 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:51.878823 23613 raft_consensus.cc:374] T bc91cdcb85a24e869a250a882927abd8 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:51.878851 23613 raft_consensus.cc:729] T bc91cdcb85a24e869a250a882927abd8 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:51.878886 23691 tablet_bootstrap.cc:654] T 0cda7f68204345cfa501efbb822135c2 P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:51.878969 23613 consensus_queue.cc:260] T bc91cdcb85a24e869a250a882927abd8 P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:51.879444 23613 ts_tablet_manager.cc:1398] T bc91cdcb85a24e869a250a882927abd8 P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:51.880199 15604 tablet_service.cc:1467] Processing CreateTablet for tablet 0cda7f68204345cfa501efbb822135c2 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=62c61e6a8f154573861f621e3546f7cf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:51.880748 23692 raft_consensus.cc:348] T bc91cdcb85a24e869a250a882927abd8 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:51.880916 23692 raft_consensus.cc:374] T bc91cdcb85a24e869a250a882927abd8 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:51.881006 23691 tablet_bootstrap.cc:492] T 0cda7f68204345cfa501efbb822135c2 P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:51.881075 23691 ts_tablet_manager.cc:1367] T 0cda7f68204345cfa501efbb822135c2 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:56:51.881124 15604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cda7f68204345cfa501efbb822135c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:51.881503 23691 raft_consensus.cc:348] T 0cda7f68204345cfa501efbb822135c2 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:51.881608 23691 raft_consensus.cc:374] T 0cda7f68204345cfa501efbb822135c2 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:51.881636 23691 raft_consensus.cc:729] T 0cda7f68204345cfa501efbb822135c2 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:51.881743 23691 consensus_queue.cc:260] T 0cda7f68204345cfa501efbb822135c2 P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:51.882025 23691 ts_tablet_manager.cc:1398] T 0cda7f68204345cfa501efbb822135c2 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:51.882539 23692 raft_consensus.cc:729] T bc91cdcb85a24e869a250a882927abd8 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:51.883035 23692 consensus_queue.cc:260] T bc91cdcb85a24e869a250a882927abd8 P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:51.884732 23692 ts_tablet_manager.cc:1398] T bc91cdcb85a24e869a250a882927abd8 P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
I20240503 13:56:51.889048 23613 tablet_bootstrap.cc:492] T 0cda7f68204345cfa501efbb822135c2 P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:51.890290 23613 tablet_bootstrap.cc:654] T 0cda7f68204345cfa501efbb822135c2 P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:51.891523 23612 tablet_bootstrap.cc:492] T 0cda7f68204345cfa501efbb822135c2 P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:51.891630 23612 ts_tablet_manager.cc:1367] T 0cda7f68204345cfa501efbb822135c2 P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.000s
I20240503 13:56:51.895776 23612 raft_consensus.cc:348] T 0cda7f68204345cfa501efbb822135c2 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:51.895920 23612 raft_consensus.cc:374] T 0cda7f68204345cfa501efbb822135c2 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:51.895954 23612 raft_consensus.cc:729] T 0cda7f68204345cfa501efbb822135c2 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:51.896077 23612 consensus_queue.cc:260] T 0cda7f68204345cfa501efbb822135c2 P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:51.896402 23612 ts_tablet_manager.cc:1398] T 0cda7f68204345cfa501efbb822135c2 P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240503 13:56:51.900688 23613 tablet_bootstrap.cc:492] T 0cda7f68204345cfa501efbb822135c2 P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:51.900806 23613 ts_tablet_manager.cc:1367] T 0cda7f68204345cfa501efbb822135c2 P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 13:56:51.901522 23613 raft_consensus.cc:348] T 0cda7f68204345cfa501efbb822135c2 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:51.901702 23613 raft_consensus.cc:374] T 0cda7f68204345cfa501efbb822135c2 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:51.901787 23613 raft_consensus.cc:729] T 0cda7f68204345cfa501efbb822135c2 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:51.901903 13032 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 13:56:51.901971 23613 consensus_queue.cc:260] T 0cda7f68204345cfa501efbb822135c2 P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:51.904876 23613 ts_tablet_manager.cc:1398] T 0cda7f68204345cfa501efbb822135c2 P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 13:56:51.907500 15604 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:56:51.908836 23693 tablet_bootstrap.cc:492] T aa97788f7cdb4caaa243a609b34f55cf P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:51.908946 23693 ts_tablet_manager.cc:1367] T aa97788f7cdb4caaa243a609b34f55cf P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.039s user 0.002s sys 0.000s
I20240503 13:56:51.909430 23693 raft_consensus.cc:348] T aa97788f7cdb4caaa243a609b34f55cf P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:51.909519 23693 raft_consensus.cc:374] T aa97788f7cdb4caaa243a609b34f55cf P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:51.909538 23693 raft_consensus.cc:729] T aa97788f7cdb4caaa243a609b34f55cf P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:51.909644 23693 consensus_queue.cc:260] T aa97788f7cdb4caaa243a609b34f55cf P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:51.909910 23693 ts_tablet_manager.cc:1398] T aa97788f7cdb4caaa243a609b34f55cf P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:51.913399 15736 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 13:56:51.921312 15867 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:56:51.924674 15472 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 13:56:51.959571 13427 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:51.961357 13163 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 13:56:51.969452 13295 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 13:56:51.977231 23473 raft_consensus.cc:482] T bc91cdcb85a24e869a250a882927abd8 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:51.977350 23473 raft_consensus.cc:504] T bc91cdcb85a24e869a250a882927abd8 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:51.977660 23473 leader_election.cc:288] T bc91cdcb85a24e869a250a882927abd8 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951), 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:51.978778 15624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc91cdcb85a24e869a250a882927abd8" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d" is_pre_election: true
I20240503 13:56:51.978909 15624 raft_consensus.cc:2454] T bc91cdcb85a24e869a250a882927abd8 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 0.
I20240503 13:56:51.978897 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc91cdcb85a24e869a250a882927abd8" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
I20240503 13:56:51.979027 13445 raft_consensus.cc:2454] T bc91cdcb85a24e869a250a882927abd8 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 0.
I20240503 13:56:51.979260 13249 leader_election.cc:302] T bc91cdcb85a24e869a250a882927abd8 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:51.979408 23397 raft_consensus.cc:2781] T bc91cdcb85a24e869a250a882927abd8 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:51.979456 23397 raft_consensus.cc:482] T bc91cdcb85a24e869a250a882927abd8 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:51.979491 23397 raft_consensus.cc:3037] T bc91cdcb85a24e869a250a882927abd8 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:51.980593 23397 raft_consensus.cc:504] T bc91cdcb85a24e869a250a882927abd8 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:51.980934 23397 leader_election.cc:288] T bc91cdcb85a24e869a250a882927abd8 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: Requested vote from peers 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951), 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:51.981280 15624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc91cdcb85a24e869a250a882927abd8" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d"
I20240503 13:56:51.981384 15624 raft_consensus.cc:3037] T bc91cdcb85a24e869a250a882927abd8 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:51.982072 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc91cdcb85a24e869a250a882927abd8" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
I20240503 13:56:51.982175 13445 raft_consensus.cc:3037] T bc91cdcb85a24e869a250a882927abd8 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:51.982426 15624 raft_consensus.cc:2454] T bc91cdcb85a24e869a250a882927abd8 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 1.
I20240503 13:56:51.982774 13249 leader_election.cc:302] T bc91cdcb85a24e869a250a882927abd8 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:51.982973 23397 raft_consensus.cc:2781] T bc91cdcb85a24e869a250a882927abd8 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:51.983047 23397 raft_consensus.cc:686] T bc91cdcb85a24e869a250a882927abd8 P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Becoming Leader. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Running, Role: LEADER
I20240503 13:56:51.983196 13445 raft_consensus.cc:2454] T bc91cdcb85a24e869a250a882927abd8 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 1.
I20240503 13:56:51.983170 23397 consensus_queue.cc:237] T bc91cdcb85a24e869a250a882927abd8 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:51.985528 12908 catalog_manager.cc:5484] T bc91cdcb85a24e869a250a882927abd8 P ba0194098aa84449ae17e8a0d4c43703 reported cstate change: term changed from 0 to 1, leader changed from <none> to ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67). New cstate: current_term: 1 leader_uuid: "ba0194098aa84449ae17e8a0d4c43703" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:51.987524 12910 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:56:51.991253 15624 raft_consensus.cc:1261] T 7f98c13e21054e6ba8a52827f91c60e8 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Refusing update from remote peer 61b887f0d40d497badeca5604c1f0546: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:51.991452 23715 raft_consensus.cc:482] T 0cda7f68204345cfa501efbb822135c2 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:51.991550 23715 raft_consensus.cc:504] T 0cda7f68204345cfa501efbb822135c2 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:51.991647 23642 consensus_queue.cc:1035] T 7f98c13e21054e6ba8a52827f91c60e8 P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:51.992058 23715 leader_election.cc:288] T 0cda7f68204345cfa501efbb822135c2 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885)
I20240503 13:56:51.992527 15758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cda7f68204345cfa501efbb822135c2" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2" is_pre_election: true
I20240503 13:56:51.992656 15758 raft_consensus.cc:2454] T 0cda7f68204345cfa501efbb822135c2 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 0.
I20240503 13:56:51.993218 15561 leader_election.cc:302] T 0cda7f68204345cfa501efbb822135c2 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 76c386bef61a4bd5a907266a5118ffe2; no voters:
I20240503 13:56:51.993367 23715 raft_consensus.cc:2781] T 0cda7f68204345cfa501efbb822135c2 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:51.993405 23715 raft_consensus.cc:482] T 0cda7f68204345cfa501efbb822135c2 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:51.993438 23715 raft_consensus.cc:3037] T 0cda7f68204345cfa501efbb822135c2 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:51.994412 23715 raft_consensus.cc:504] T 0cda7f68204345cfa501efbb822135c2 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:51.994722 23715 leader_election.cc:288] T 0cda7f68204345cfa501efbb822135c2 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 election: Requested vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885)
I20240503 13:56:51.996372 23719 tablet.cc:1722] T 7f98c13e21054e6ba8a52827f91c60e8 P 61b887f0d40d497badeca5604c1f0546: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:51.996408 12910 catalog_manager.cc:5576] TS 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951) does not have the latest schema for tablet 7f98c13e21054e6ba8a52827f91c60e8 (table rebalancer_dynamic_table [id=75c555d294b9416b8478d1b0898b2cba]). Expected version 1 got 0
I20240503 13:56:51.996729 23719 tablet.cc:1620] T 7f98c13e21054e6ba8a52827f91c60e8 P 61b887f0d40d497badeca5604c1f0546: MemRowSet was empty: no flush needed.
I20240503 13:56:51.996830 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cda7f68204345cfa501efbb822135c2" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
I20240503 13:56:51.996948 13447 raft_consensus.cc:3037] T 0cda7f68204345cfa501efbb822135c2 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:51.997931 13447 raft_consensus.cc:2454] T 0cda7f68204345cfa501efbb822135c2 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 1.
I20240503 13:56:51.998232 23720 tablet.cc:1722] T 7f98c13e21054e6ba8a52827f91c60e8 P 009f849ee7b54c80a3f5264695047e4d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:51.998556 23720 tablet.cc:1620] T 7f98c13e21054e6ba8a52827f91c60e8 P 009f849ee7b54c80a3f5264695047e4d: MemRowSet was empty: no flush needed.
I20240503 13:56:51.998778 15757 raft_consensus.cc:1261] T 7f98c13e21054e6ba8a52827f91c60e8 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Refusing update from remote peer 61b887f0d40d497badeca5604c1f0546: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:51.999248 23642 consensus_queue.cc:1035] T 7f98c13e21054e6ba8a52827f91c60e8 P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:52.001024 15758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cda7f68204345cfa501efbb822135c2" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2"
I20240503 13:56:52.001148 15758 raft_consensus.cc:3037] T 0cda7f68204345cfa501efbb822135c2 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:52.002022 15758 raft_consensus.cc:2454] T 0cda7f68204345cfa501efbb822135c2 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 1.
I20240503 13:56:52.002279 15561 leader_election.cc:302] T 0cda7f68204345cfa501efbb822135c2 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 76c386bef61a4bd5a907266a5118ffe2; no voters:
I20240503 13:56:52.002521 23621 raft_consensus.cc:2781] T 0cda7f68204345cfa501efbb822135c2 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:52.002617 23621 raft_consensus.cc:686] T 0cda7f68204345cfa501efbb822135c2 P 009f849ee7b54c80a3f5264695047e4d [term 1 LEADER]: Becoming Leader. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Running, Role: LEADER
I20240503 13:56:52.002755 23621 consensus_queue.cc:237] T 0cda7f68204345cfa501efbb822135c2 P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:52.005488 23614 mvcc.cc:205] Tried to move back new op lower bound from 7023593930712383488 to 7023593930172358656. Current Snapshot: MvccSnapshot[applied={T|T < 7023593930712383488}]
I20240503 13:56:52.005584 12910 catalog_manager.cc:5484] T 0cda7f68204345cfa501efbb822135c2 P 009f849ee7b54c80a3f5264695047e4d reported cstate change: term changed from 0 to 1, leader changed from <none> to 009f849ee7b54c80a3f5264695047e4d (127.6.98.70). New cstate: current_term: 1 leader_uuid: "009f849ee7b54c80a3f5264695047e4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:52.006183 23723 tablet.cc:1722] T 7f98c13e21054e6ba8a52827f91c60e8 P 76c386bef61a4bd5a907266a5118ffe2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:52.006943 23723 tablet.cc:1620] T 7f98c13e21054e6ba8a52827f91c60e8 P 76c386bef61a4bd5a907266a5118ffe2: MemRowSet was empty: no flush needed.
I20240503 13:56:52.007938 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cda7f68204345cfa501efbb822135c2" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
I20240503 13:56:52.008071 13445 raft_consensus.cc:2362] T 0cda7f68204345cfa501efbb822135c2 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 1. Re-sending same reply.
I20240503 13:56:52.008739 12908 catalog_manager.cc:5576] TS 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885) does not have the latest schema for tablet 7f98c13e21054e6ba8a52827f91c60e8 (table rebalancer_dynamic_table [id=75c555d294b9416b8478d1b0898b2cba]). Expected version 1 got 0
I20240503 13:56:52.008898 12906 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=75c555d294b9416b8478d1b0898b2cba] alter complete (version 1)
I20240503 13:56:52.032580 23002 raft_consensus.cc:482] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:52.032940 23002 raft_consensus.cc:504] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:52.033385 23002 leader_election.cc:288] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:52.033905 15889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97788f7cdb4caaa243a609b34f55cf" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7" is_pre_election: true
I20240503 13:56:52.033977 13185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97788f7cdb4caaa243a609b34f55cf" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770" is_pre_election: true
I20240503 13:56:52.034049 15889 raft_consensus.cc:2454] T aa97788f7cdb4caaa243a609b34f55cf P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 0.
I20240503 13:56:52.034127 13185 raft_consensus.cc:2454] T aa97788f7cdb4caaa243a609b34f55cf P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 0.
I20240503 13:56:52.034404 13249 leader_election.cc:302] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba0194098aa84449ae17e8a0d4c43703, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:52.034598 23002 raft_consensus.cc:2781] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:52.034657 23002 raft_consensus.cc:482] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:52.034700 23002 raft_consensus.cc:3037] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:52.035796 23002 raft_consensus.cc:504] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:52.036144 23002 leader_election.cc:288] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: Requested vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:52.036444 15889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97788f7cdb4caaa243a609b34f55cf" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7"
I20240503 13:56:52.036588 15889 raft_consensus.cc:3037] T aa97788f7cdb4caaa243a609b34f55cf P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:52.036685 13182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97788f7cdb4caaa243a609b34f55cf" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770"
I20240503 13:56:52.036806 13182 raft_consensus.cc:3037] T aa97788f7cdb4caaa243a609b34f55cf P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:52.037566 15889 raft_consensus.cc:2454] T aa97788f7cdb4caaa243a609b34f55cf P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 1.
I20240503 13:56:52.037883 13182 raft_consensus.cc:2454] T aa97788f7cdb4caaa243a609b34f55cf P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 1.
I20240503 13:56:52.037870 13249 leader_election.cc:302] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba0194098aa84449ae17e8a0d4c43703, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:52.038903 23002 raft_consensus.cc:2781] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:52.039098 23002 raft_consensus.cc:686] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Becoming Leader. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Running, Role: LEADER
I20240503 13:56:52.039235 23002 consensus_queue.cc:237] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:52.042281 12908 catalog_manager.cc:5484] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 reported cstate change: term changed from 0 to 1, leader changed from <none> to ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67). New cstate: current_term: 1 leader_uuid: "ba0194098aa84449ae17e8a0d4c43703" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } }
I20240503 13:56:52.047901 13445 consensus_queue.cc:237] T b34b432bb82e465f87a6a3b05d7adfd2 P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:52.049187 15492 raft_consensus.cc:1261] T b34b432bb82e465f87a6a3b05d7adfd2 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Refusing update from remote peer 7690ee79577c45e3a2c8f8b552c0bff8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:56:52.049373 13052 raft_consensus.cc:1261] T b34b432bb82e465f87a6a3b05d7adfd2 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Refusing update from remote peer 7690ee79577c45e3a2c8f8b552c0bff8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:56:52.049558 23741 consensus_queue.cc:1035] T b34b432bb82e465f87a6a3b05d7adfd2 P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:52.049782 23172 consensus_queue.cc:1035] T b34b432bb82e465f87a6a3b05d7adfd2 P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:52.053300 23741 raft_consensus.cc:2932] T b34b432bb82e465f87a6a3b05d7adfd2 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:52.055195 13052 raft_consensus.cc:2932] T b34b432bb82e465f87a6a3b05d7adfd2 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:52.055536 15492 raft_consensus.cc:2932] T b34b432bb82e465f87a6a3b05d7adfd2 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:52.056324 12908 catalog_manager.cc:5484] T b34b432bb82e465f87a6a3b05d7adfd2 P 7690ee79577c45e3a2c8f8b552c0bff8 reported cstate change: config changed from index 4 to 5, VOTER ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) evicted. New cstate: current_term: 1 leader_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:52.061004 13295 tablet_service.cc:1514] Processing DeleteTablet for tablet b34b432bb82e465f87a6a3b05d7adfd2 with delete_type TABLET_DATA_TOMBSTONED (TS ba0194098aa84449ae17e8a0d4c43703 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:33250
I20240503 13:56:52.061388 23745 tablet_replica.cc:330] T b34b432bb82e465f87a6a3b05d7adfd2 P ba0194098aa84449ae17e8a0d4c43703: stopping tablet replica
I20240503 13:56:52.061524 23745 raft_consensus.cc:2229] T b34b432bb82e465f87a6a3b05d7adfd2 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:52.061879 23745 raft_consensus.cc:2258] T b34b432bb82e465f87a6a3b05d7adfd2 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:52.062551 23745 ts_tablet_manager.cc:1875] T b34b432bb82e465f87a6a3b05d7adfd2 P ba0194098aa84449ae17e8a0d4c43703: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:56:52.069598 23745 ts_tablet_manager.cc:1888] T b34b432bb82e465f87a6a3b05d7adfd2 P ba0194098aa84449ae17e8a0d4c43703: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 13:56:52.069684 23745 log.cc:1196] T b34b432bb82e465f87a6a3b05d7adfd2 P ba0194098aa84449ae17e8a0d4c43703: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/wal/wals/b34b432bb82e465f87a6a3b05d7adfd2
I20240503 13:56:52.070209 12899 catalog_manager.cc:4830] TS ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279): tablet b34b432bb82e465f87a6a3b05d7adfd2 (table rebalance_test_table_15 [id=2bf18d88eb674480901ad018abd82ad1]) successfully deleted
I20240503 13:56:52.112090 12906 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:56:52.120220 23719 tablet.cc:1722] T 7f98c13e21054e6ba8a52827f91c60e8 P 61b887f0d40d497badeca5604c1f0546: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:52.120606 23719 tablet.cc:1620] T 7f98c13e21054e6ba8a52827f91c60e8 P 61b887f0d40d497badeca5604c1f0546: MemRowSet was empty: no flush needed.
I20240503 13:56:52.121609 23723 tablet.cc:1722] T 7f98c13e21054e6ba8a52827f91c60e8 P 76c386bef61a4bd5a907266a5118ffe2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:52.121913 23723 tablet.cc:1620] T 7f98c13e21054e6ba8a52827f91c60e8 P 76c386bef61a4bd5a907266a5118ffe2: MemRowSet was empty: no flush needed.
I20240503 13:56:52.124011 23720 tablet.cc:1722] T 7f98c13e21054e6ba8a52827f91c60e8 P 009f849ee7b54c80a3f5264695047e4d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:52.124323 23720 tablet.cc:1620] T 7f98c13e21054e6ba8a52827f91c60e8 P 009f849ee7b54c80a3f5264695047e4d: MemRowSet was empty: no flush needed.
I20240503 13:56:52.132292 12910 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=75c555d294b9416b8478d1b0898b2cba] alter complete (version 2)
I20240503 13:56:52.207080 13032 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 13:56:52.207190 15604 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 13:56:52.208811 15472 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 13:56:52.231221 13295 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 13:56:52.231837 13427 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:52.239416 13163 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 13:56:52.240680 15736 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 13:56:52.241941 12905 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:56:52.252112 15867 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:56:52.260462 23719 tablet.cc:1722] T 7f98c13e21054e6ba8a52827f91c60e8 P 61b887f0d40d497badeca5604c1f0546: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:52.265586 23723 tablet.cc:1722] T 7f98c13e21054e6ba8a52827f91c60e8 P 76c386bef61a4bd5a907266a5118ffe2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:52.267202 12905 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table_new_name [id=75c555d294b9416b8478d1b0898b2cba] alter complete (version 3)
I20240503 13:56:52.269235 23720 tablet.cc:1722] T 7f98c13e21054e6ba8a52827f91c60e8 P 009f849ee7b54c80a3f5264695047e4d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:52.332324 15757 consensus_queue.cc:237] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:52.333786 15492 raft_consensus.cc:1261] T 604875eff3594876af54e9126057bd7f P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Refusing update from remote peer 76c386bef61a4bd5a907266a5118ffe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:56:52.334142 23686 consensus_queue.cc:1035] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:56:52.334715 15889 raft_consensus.cc:1261] T 604875eff3594876af54e9126057bd7f P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Refusing update from remote peer 76c386bef61a4bd5a907266a5118ffe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:56:52.335146 23686 consensus_queue.cc:1035] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:56:52.336308 23131 raft_consensus.cc:2932] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 009f849ee7b54c80a3f5264695047e4d (127.6.98.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:52.336694 15493 raft_consensus.cc:2932] T 604875eff3594876af54e9126057bd7f P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 009f849ee7b54c80a3f5264695047e4d (127.6.98.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:52.339578 12905 catalog_manager.cc:5484] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 reported cstate change: config changed from index 2 to 3, VOTER 009f849ee7b54c80a3f5264695047e4d (127.6.98.70) evicted. New cstate: current_term: 1 leader_uuid: "76c386bef61a4bd5a907266a5118ffe2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:52.341296 15889 raft_consensus.cc:2932] T 604875eff3594876af54e9126057bd7f P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 009f849ee7b54c80a3f5264695047e4d (127.6.98.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:52.345743 15604 tablet_service.cc:1514] Processing DeleteTablet for tablet 604875eff3594876af54e9126057bd7f with delete_type TABLET_DATA_TOMBSTONED (TS 009f849ee7b54c80a3f5264695047e4d not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:60718
I20240503 13:56:52.346097 23827 tablet_replica.cc:330] T 604875eff3594876af54e9126057bd7f P 009f849ee7b54c80a3f5264695047e4d: stopping tablet replica
I20240503 13:56:52.346246 23827 raft_consensus.cc:2229] T 604875eff3594876af54e9126057bd7f P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:52.346362 23827 raft_consensus.cc:2258] T 604875eff3594876af54e9126057bd7f P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:52.347843 23827 ts_tablet_manager.cc:1875] T 604875eff3594876af54e9126057bd7f P 009f849ee7b54c80a3f5264695047e4d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:56:52.350571 23827 ts_tablet_manager.cc:1888] T 604875eff3594876af54e9126057bd7f P 009f849ee7b54c80a3f5264695047e4d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 13:56:52.350711 23827 log.cc:1196] T 604875eff3594876af54e9126057bd7f P 009f849ee7b54c80a3f5264695047e4d: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/wal/wals/604875eff3594876af54e9126057bd7f
I20240503 13:56:52.351167 12896 catalog_manager.cc:4830] TS 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951): tablet 604875eff3594876af54e9126057bd7f (table rebalancer_extra_table_6 [id=1161fd279f6d435887536e9c124d251a]) successfully deleted
I20240503 13:56:52.362170 13182 consensus_queue.cc:237] T 7239f94a7c88433aa6062390a54e7562 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:52.363703 13445 raft_consensus.cc:1261] T 7239f94a7c88433aa6062390a54e7562 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer 8ccf5a4ac450414c9d3579bd5800b770: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 13:56:52.363827 13118 consensus_peers.cc:485] T 7239f94a7c88433aa6062390a54e7562 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951): Couldn't send request to peer 009f849ee7b54c80a3f5264695047e4d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7239f94a7c88433aa6062390a54e7562. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:52.364221 13316 raft_consensus.cc:1261] T 7239f94a7c88433aa6062390a54e7562 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Refusing update from remote peer 8ccf5a4ac450414c9d3579bd5800b770: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:56:52.364665 23839 consensus_queue.cc:1035] T 7239f94a7c88433aa6062390a54e7562 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:52.364886 23306 consensus_queue.cc:1035] T 7239f94a7c88433aa6062390a54e7562 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:52.366091 23397 consensus_queue.cc:1035] T bc91cdcb85a24e869a250a882927abd8 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:52.367095 23306 raft_consensus.cc:2932] T 7239f94a7c88433aa6062390a54e7562 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 009f849ee7b54c80a3f5264695047e4d (127.6.98.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:52.369598 13316 raft_consensus.cc:2932] T 7239f94a7c88433aa6062390a54e7562 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 009f849ee7b54c80a3f5264695047e4d (127.6.98.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:52.371614 12908 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:52.371749 12908 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:52.373473 13447 raft_consensus.cc:2932] T 7239f94a7c88433aa6062390a54e7562 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 009f849ee7b54c80a3f5264695047e4d (127.6.98.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:52.374153 12905 catalog_manager.cc:5484] T 7239f94a7c88433aa6062390a54e7562 P 8ccf5a4ac450414c9d3579bd5800b770 reported cstate change: config changed from index -1 to 4, VOTER 009f849ee7b54c80a3f5264695047e4d (127.6.98.70) added. New cstate: current_term: 1 leader_uuid: "8ccf5a4ac450414c9d3579bd5800b770" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:52.375181 12908 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Sending DeleteTablet for 3 replicas of tablet 7f98c13e21054e6ba8a52827f91c60e8
I20240503 13:56:52.376443 15736 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f98c13e21054e6ba8a52827f91c60e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:52 UTC) from {username='slave'} at 127.0.0.1:44984
I20240503 13:56:52.376724 15472 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f98c13e21054e6ba8a52827f91c60e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:52 UTC) from {username='slave'} at 127.0.0.1:58758
I20240503 13:56:52.376926 15604 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f98c13e21054e6ba8a52827f91c60e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:52 UTC) from {username='slave'} at 127.0.0.1:60718
I20240503 13:56:52.377318 23827 tablet_replica.cc:330] T 7f98c13e21054e6ba8a52827f91c60e8 P 009f849ee7b54c80a3f5264695047e4d: stopping tablet replica
I20240503 13:56:52.377470 23827 raft_consensus.cc:2229] T 7f98c13e21054e6ba8a52827f91c60e8 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:52.377580 23845 tablet_replica.cc:330] T 7f98c13e21054e6ba8a52827f91c60e8 P 61b887f0d40d497badeca5604c1f0546: stopping tablet replica
I20240503 13:56:52.377584 23827 raft_consensus.cc:2258] T 7f98c13e21054e6ba8a52827f91c60e8 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:52.377709 23845 raft_consensus.cc:2229] T 7f98c13e21054e6ba8a52827f91c60e8 P 61b887f0d40d497badeca5604c1f0546 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:56:52.377899 23845 raft_consensus.cc:2258] T 7f98c13e21054e6ba8a52827f91c60e8 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:52.378559 23827 ts_tablet_manager.cc:1875] T 7f98c13e21054e6ba8a52827f91c60e8 P 009f849ee7b54c80a3f5264695047e4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:52.379143 23844 tablet_replica.cc:330] T 7f98c13e21054e6ba8a52827f91c60e8 P 76c386bef61a4bd5a907266a5118ffe2: stopping tablet replica
I20240503 13:56:52.379273 23844 raft_consensus.cc:2229] T 7f98c13e21054e6ba8a52827f91c60e8 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:52.379542 23844 raft_consensus.cc:2258] T 7f98c13e21054e6ba8a52827f91c60e8 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:52.379748 23397 consensus_queue.cc:1035] T bc91cdcb85a24e869a250a882927abd8 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:52.380089 23844 ts_tablet_manager.cc:1875] T 7f98c13e21054e6ba8a52827f91c60e8 P 76c386bef61a4bd5a907266a5118ffe2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:52.383431 23844 ts_tablet_manager.cc:1888] T 7f98c13e21054e6ba8a52827f91c60e8 P 76c386bef61a4bd5a907266a5118ffe2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:52.383562 23844 log.cc:1196] T 7f98c13e21054e6ba8a52827f91c60e8 P 76c386bef61a4bd5a907266a5118ffe2: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/wal/wals/7f98c13e21054e6ba8a52827f91c60e8
I20240503 13:56:52.383877 23844 ts_tablet_manager.cc:1909] T 7f98c13e21054e6ba8a52827f91c60e8 P 76c386bef61a4bd5a907266a5118ffe2: Deleting consensus metadata
I20240503 13:56:52.385602 12899 catalog_manager.cc:4830] TS 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885): tablet 7f98c13e21054e6ba8a52827f91c60e8 (table rebalancer_dynamic_table_new_name [id=75c555d294b9416b8478d1b0898b2cba]) successfully deleted
I20240503 13:56:52.386406 23827 ts_tablet_manager.cc:1888] T 7f98c13e21054e6ba8a52827f91c60e8 P 009f849ee7b54c80a3f5264695047e4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:52.386485 23827 log.cc:1196] T 7f98c13e21054e6ba8a52827f91c60e8 P 009f849ee7b54c80a3f5264695047e4d: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/wal/wals/7f98c13e21054e6ba8a52827f91c60e8
I20240503 13:56:52.386762 23827 ts_tablet_manager.cc:1909] T 7f98c13e21054e6ba8a52827f91c60e8 P 009f849ee7b54c80a3f5264695047e4d: Deleting consensus metadata
I20240503 13:56:52.386898 23845 ts_tablet_manager.cc:1875] T 7f98c13e21054e6ba8a52827f91c60e8 P 61b887f0d40d497badeca5604c1f0546: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:52.387507 12896 catalog_manager.cc:4830] TS 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951): tablet 7f98c13e21054e6ba8a52827f91c60e8 (table rebalancer_dynamic_table_new_name [id=75c555d294b9416b8478d1b0898b2cba]) successfully deleted
I20240503 13:56:52.389969 23845 ts_tablet_manager.cc:1888] T 7f98c13e21054e6ba8a52827f91c60e8 P 61b887f0d40d497badeca5604c1f0546: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:52.390038 23845 log.cc:1196] T 7f98c13e21054e6ba8a52827f91c60e8 P 61b887f0d40d497badeca5604c1f0546: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/wal/wals/7f98c13e21054e6ba8a52827f91c60e8
I20240503 13:56:52.390250 23845 ts_tablet_manager.cc:1909] T 7f98c13e21054e6ba8a52827f91c60e8 P 61b887f0d40d497badeca5604c1f0546: Deleting consensus metadata
I20240503 13:56:52.390973 12899 catalog_manager.cc:4830] TS 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153): tablet 7f98c13e21054e6ba8a52827f91c60e8 (table rebalancer_dynamic_table_new_name [id=75c555d294b9416b8478d1b0898b2cba]) successfully deleted
I20240503 13:56:52.477186 12907 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:56:52.484632 15867 tablet_service.cc:1467] Processing CreateTablet for tablet f64b8b4b4c4c4413b2da874c8751710e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=473ab6f00fab4abb9fb40247a66eb620]), partition=
I20240503 13:56:52.484925 15867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f64b8b4b4c4c4413b2da874c8751710e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:52.487316 23890 tablet_bootstrap.cc:492] T f64b8b4b4c4c4413b2da874c8751710e P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:52.487442 13032 tablet_service.cc:1467] Processing CreateTablet for tablet f64b8b4b4c4c4413b2da874c8751710e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=473ab6f00fab4abb9fb40247a66eb620]), partition=
I20240503 13:56:52.487672 13032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f64b8b4b4c4c4413b2da874c8751710e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:52.488230 23890 tablet_bootstrap.cc:654] T f64b8b4b4c4c4413b2da874c8751710e P e47105a0f46e4511b7112e571688a4c7: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:52.490078 23892 tablet_bootstrap.cc:492] T f64b8b4b4c4c4413b2da874c8751710e P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:52.490152 23890 tablet_bootstrap.cc:492] T f64b8b4b4c4c4413b2da874c8751710e P e47105a0f46e4511b7112e571688a4c7: No bootstrap required, opened a new log
I20240503 13:56:52.490595 23890 ts_tablet_manager.cc:1367] T f64b8b4b4c4c4413b2da874c8751710e P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:56:52.491080 13295 tablet_service.cc:1467] Processing CreateTablet for tablet f64b8b4b4c4c4413b2da874c8751710e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=473ab6f00fab4abb9fb40247a66eb620]), partition=
I20240503 13:56:52.491320 13295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f64b8b4b4c4c4413b2da874c8751710e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:52.491597 23890 raft_consensus.cc:348] T f64b8b4b4c4c4413b2da874c8751710e P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:52.491714 23890 raft_consensus.cc:374] T f64b8b4b4c4c4413b2da874c8751710e P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:52.491744 23890 raft_consensus.cc:729] T f64b8b4b4c4c4413b2da874c8751710e P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:52.492067 23890 consensus_queue.cc:260] T f64b8b4b4c4c4413b2da874c8751710e P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:52.493702 23890 ts_tablet_manager.cc:1398] T f64b8b4b4c4c4413b2da874c8751710e P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:56:52.500190 23892 tablet_bootstrap.cc:654] T f64b8b4b4c4c4413b2da874c8751710e P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:52.501416 23893 tablet_bootstrap.cc:492] T f64b8b4b4c4c4413b2da874c8751710e P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:52.503171 23893 tablet_bootstrap.cc:654] T f64b8b4b4c4c4413b2da874c8751710e P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:52.504798 23892 tablet_bootstrap.cc:492] T f64b8b4b4c4c4413b2da874c8751710e P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:52.504904 23892 ts_tablet_manager.cc:1367] T f64b8b4b4c4c4413b2da874c8751710e P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.001s
I20240503 13:56:52.505371 23892 raft_consensus.cc:348] T f64b8b4b4c4c4413b2da874c8751710e P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:52.505590 23892 raft_consensus.cc:374] T f64b8b4b4c4c4413b2da874c8751710e P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:52.505669 23892 raft_consensus.cc:729] T f64b8b4b4c4c4413b2da874c8751710e P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:52.505846 23892 consensus_queue.cc:260] T f64b8b4b4c4c4413b2da874c8751710e P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:52.508159 23621 consensus_queue.cc:1035] T 0cda7f68204345cfa501efbb822135c2 P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:52.508452 23621 consensus_queue.cc:1035] T 0cda7f68204345cfa501efbb822135c2 P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:52.509007 23892 ts_tablet_manager.cc:1398] T f64b8b4b4c4c4413b2da874c8751710e P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 13:56:52.511070 23893 tablet_bootstrap.cc:492] T f64b8b4b4c4c4413b2da874c8751710e P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:52.511224 23893 ts_tablet_manager.cc:1367] T f64b8b4b4c4c4413b2da874c8751710e P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 13:56:52.511765 23893 raft_consensus.cc:348] T f64b8b4b4c4c4413b2da874c8751710e P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:52.513803 23893 raft_consensus.cc:374] T f64b8b4b4c4c4413b2da874c8751710e P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:52.513886 23893 raft_consensus.cc:729] T f64b8b4b4c4c4413b2da874c8751710e P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:52.514097 23893 consensus_queue.cc:260] T f64b8b4b4c4c4413b2da874c8751710e P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:52.515475 13427 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:52.518424 23893 ts_tablet_manager.cc:1398] T f64b8b4b4c4c4413b2da874c8751710e P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240503 13:56:52.527916 13032 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 13:56:52.529619 15604 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 13:56:52.532409 15472 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 13:56:52.548345 13295 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 13:56:52.548815 15736 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 13:56:52.560094 15867 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:56:52.560400 13163 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 13:56:52.579411 12906 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_extra_table_60"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:56:52.589560 15604 tablet_service.cc:1467] Processing CreateTablet for tablet 8c5e25679b894ac4a86c2248c7f4f1bb (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=fab66811f3824bb4bf6ecd5007ded85e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:52.589886 15604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c5e25679b894ac4a86c2248c7f4f1bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:52.589911 15736 tablet_service.cc:1467] Processing CreateTablet for tablet 6c28bbb731834bf59af42d21a55fdffe (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=fab66811f3824bb4bf6ecd5007ded85e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:52.589921 13163 tablet_service.cc:1467] Processing CreateTablet for tablet 8c5e25679b894ac4a86c2248c7f4f1bb (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=fab66811f3824bb4bf6ecd5007ded85e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:52.590188 13163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c5e25679b894ac4a86c2248c7f4f1bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:52.590199 15736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c28bbb731834bf59af42d21a55fdffe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:52.590237 13427 tablet_service.cc:1467] Processing CreateTablet for tablet 8c5e25679b894ac4a86c2248c7f4f1bb (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=fab66811f3824bb4bf6ecd5007ded85e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:52.590500 13427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c5e25679b894ac4a86c2248c7f4f1bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:52.592422 15605 tablet_service.cc:1467] Processing CreateTablet for tablet 6c28bbb731834bf59af42d21a55fdffe (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=fab66811f3824bb4bf6ecd5007ded85e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:52.592689 15605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c28bbb731834bf59af42d21a55fdffe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:52.593029 15472 tablet_service.cc:1467] Processing CreateTablet for tablet 39b50d0b7d934e47ac61410e8173033b (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=fab66811f3824bb4bf6ecd5007ded85e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:52.593272 15472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39b50d0b7d934e47ac61410e8173033b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:52.595036 15604 tablet_service.cc:1467] Processing CreateTablet for tablet 39b50d0b7d934e47ac61410e8173033b (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=fab66811f3824bb4bf6ecd5007ded85e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:52.595265 15604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39b50d0b7d934e47ac61410e8173033b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:52.596304 15736 tablet_service.cc:1467] Processing CreateTablet for tablet 39b50d0b7d934e47ac61410e8173033b (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=fab66811f3824bb4bf6ecd5007ded85e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:52.596598 15736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39b50d0b7d934e47ac61410e8173033b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:52.601892 13295 tablet_service.cc:1467] Processing CreateTablet for tablet 6c28bbb731834bf59af42d21a55fdffe (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=fab66811f3824bb4bf6ecd5007ded85e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:52.602154 13295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c28bbb731834bf59af42d21a55fdffe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:52.604383 23893 tablet_bootstrap.cc:492] T 6c28bbb731834bf59af42d21a55fdffe P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:52.604697 23909 tablet_bootstrap.cc:492] T 8c5e25679b894ac4a86c2248c7f4f1bb P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:52.604871 23912 tablet_bootstrap.cc:492] T 8c5e25679b894ac4a86c2248c7f4f1bb P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:52.605362 23893 tablet_bootstrap.cc:654] T 6c28bbb731834bf59af42d21a55fdffe P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:52.605790 23909 tablet_bootstrap.cc:654] T 8c5e25679b894ac4a86c2248c7f4f1bb P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:52.608000 23912 tablet_bootstrap.cc:654] T 8c5e25679b894ac4a86c2248c7f4f1bb P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:52.608718 23911 tablet_bootstrap.cc:492] T 6c28bbb731834bf59af42d21a55fdffe P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:52.610157 23910 tablet_bootstrap.cc:492] T 8c5e25679b894ac4a86c2248c7f4f1bb P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:52.610334 23913 tablet_bootstrap.cc:492] T 39b50d0b7d934e47ac61410e8173033b P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:52.613088 23911 tablet_bootstrap.cc:654] T 6c28bbb731834bf59af42d21a55fdffe P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:52.613562 23893 tablet_bootstrap.cc:492] T 6c28bbb731834bf59af42d21a55fdffe P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:52.613668 23893 ts_tablet_manager.cc:1367] T 6c28bbb731834bf59af42d21a55fdffe P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 13:56:52.614063 23913 tablet_bootstrap.cc:654] T 39b50d0b7d934e47ac61410e8173033b P 61b887f0d40d497badeca5604c1f0546: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:52.614308 23910 tablet_bootstrap.cc:654] T 8c5e25679b894ac4a86c2248c7f4f1bb P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:52.614306 23893 raft_consensus.cc:348] T 6c28bbb731834bf59af42d21a55fdffe P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:52.614434 23893 raft_consensus.cc:374] T 6c28bbb731834bf59af42d21a55fdffe P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:52.614471 23893 raft_consensus.cc:729] T 6c28bbb731834bf59af42d21a55fdffe P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:52.614589 23893 consensus_queue.cc:260] T 6c28bbb731834bf59af42d21a55fdffe P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:52.614751 23909 tablet_bootstrap.cc:492] T 8c5e25679b894ac4a86c2248c7f4f1bb P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:52.614840 23909 ts_tablet_manager.cc:1367] T 8c5e25679b894ac4a86c2248c7f4f1bb P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 13:56:52.614863 23893 ts_tablet_manager.cc:1398] T 6c28bbb731834bf59af42d21a55fdffe P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:52.615284 23909 raft_consensus.cc:348] T 8c5e25679b894ac4a86c2248c7f4f1bb P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:52.615381 23909 raft_consensus.cc:374] T 8c5e25679b894ac4a86c2248c7f4f1bb P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:52.615410 23909 raft_consensus.cc:729] T 8c5e25679b894ac4a86c2248c7f4f1bb P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:52.615537 23909 consensus_queue.cc:260] T 8c5e25679b894ac4a86c2248c7f4f1bb P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:52.615806 23910 tablet_bootstrap.cc:492] T 8c5e25679b894ac4a86c2248c7f4f1bb P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:52.615813 23909 ts_tablet_manager.cc:1398] T 8c5e25679b894ac4a86c2248c7f4f1bb P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:52.616011 23909 tablet_bootstrap.cc:492] T 6c28bbb731834bf59af42d21a55fdffe P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:52.616925 23909 tablet_bootstrap.cc:654] T 6c28bbb731834bf59af42d21a55fdffe P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:52.618613 23912 tablet_bootstrap.cc:492] T 8c5e25679b894ac4a86c2248c7f4f1bb P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:52.618705 23912 ts_tablet_manager.cc:1367] T 8c5e25679b894ac4a86c2248c7f4f1bb P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240503 13:56:52.619128 23912 raft_consensus.cc:348] T 8c5e25679b894ac4a86c2248c7f4f1bb P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:52.619238 23912 raft_consensus.cc:374] T 8c5e25679b894ac4a86c2248c7f4f1bb P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:52.619266 23912 raft_consensus.cc:729] T 8c5e25679b894ac4a86c2248c7f4f1bb P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:52.619369 23912 consensus_queue.cc:260] T 8c5e25679b894ac4a86c2248c7f4f1bb P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:52.619623 23912 ts_tablet_manager.cc:1398] T 8c5e25679b894ac4a86c2248c7f4f1bb P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:52.615886 23910 ts_tablet_manager.cc:1367] T 8c5e25679b894ac4a86c2248c7f4f1bb P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20240503 13:56:52.621572 23910 raft_consensus.cc:348] T 8c5e25679b894ac4a86c2248c7f4f1bb P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:52.622601 23910 raft_consensus.cc:374] T 8c5e25679b894ac4a86c2248c7f4f1bb P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:52.623260 23911 tablet_bootstrap.cc:492] T 6c28bbb731834bf59af42d21a55fdffe P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:52.623359 23911 ts_tablet_manager.cc:1367] T 6c28bbb731834bf59af42d21a55fdffe P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240503 13:56:52.623855 23911 raft_consensus.cc:348] T 6c28bbb731834bf59af42d21a55fdffe P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:52.623973 23911 raft_consensus.cc:374] T 6c28bbb731834bf59af42d21a55fdffe P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:52.624045 23910 raft_consensus.cc:729] T 8c5e25679b894ac4a86c2248c7f4f1bb P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:52.624280 23911 raft_consensus.cc:729] T 6c28bbb731834bf59af42d21a55fdffe P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:52.624413 23911 consensus_queue.cc:260] T 6c28bbb731834bf59af42d21a55fdffe P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:52.624191 23910 consensus_queue.cc:260] T 8c5e25679b894ac4a86c2248c7f4f1bb P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:52.624826 23910 ts_tablet_manager.cc:1398] T 8c5e25679b894ac4a86c2248c7f4f1bb P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240503 13:56:52.626258 23913 tablet_bootstrap.cc:492] T 39b50d0b7d934e47ac61410e8173033b P 61b887f0d40d497badeca5604c1f0546: No bootstrap required, opened a new log
I20240503 13:56:52.626277 23909 tablet_bootstrap.cc:492] T 6c28bbb731834bf59af42d21a55fdffe P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:52.626360 23909 ts_tablet_manager.cc:1367] T 6c28bbb731834bf59af42d21a55fdffe P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20240503 13:56:52.626415 23913 ts_tablet_manager.cc:1367] T 39b50d0b7d934e47ac61410e8173033b P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240503 13:56:52.626822 23909 raft_consensus.cc:348] T 6c28bbb731834bf59af42d21a55fdffe P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:52.626921 23909 raft_consensus.cc:374] T 6c28bbb731834bf59af42d21a55fdffe P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:52.626950 23909 raft_consensus.cc:729] T 6c28bbb731834bf59af42d21a55fdffe P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:52.626864 23913 raft_consensus.cc:348] T 39b50d0b7d934e47ac61410e8173033b P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:52.627017 23913 raft_consensus.cc:374] T 39b50d0b7d934e47ac61410e8173033b P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:52.627044 23913 raft_consensus.cc:729] T 39b50d0b7d934e47ac61410e8173033b P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:52.627063 23909 consensus_queue.cc:260] T 6c28bbb731834bf59af42d21a55fdffe P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:52.627159 23913 consensus_queue.cc:260] T 39b50d0b7d934e47ac61410e8173033b P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:52.627321 23909 ts_tablet_manager.cc:1398] T 6c28bbb731834bf59af42d21a55fdffe P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:52.627492 23909 tablet_bootstrap.cc:492] T 39b50d0b7d934e47ac61410e8173033b P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:52.628427 23909 tablet_bootstrap.cc:654] T 39b50d0b7d934e47ac61410e8173033b P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:52.630255 23839 raft_consensus.cc:482] T 8c5e25679b894ac4a86c2248c7f4f1bb P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:52.632920 23839 raft_consensus.cc:504] T 8c5e25679b894ac4a86c2248c7f4f1bb P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:52.633098 23913 ts_tablet_manager.cc:1398] T 39b50d0b7d934e47ac61410e8173033b P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240503 13:56:52.632712 23911 ts_tablet_manager.cc:1398] T 6c28bbb731834bf59af42d21a55fdffe P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.009s user 0.000s sys 0.001s
I20240503 13:56:52.633317 23839 leader_election.cc:288] T 8c5e25679b894ac4a86c2248c7f4f1bb P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951)
I20240503 13:56:52.633919 15624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c5e25679b894ac4a86c2248c7f4f1bb" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d" is_pre_election: true
I20240503 13:56:52.634107 15624 raft_consensus.cc:2454] T 8c5e25679b894ac4a86c2248c7f4f1bb P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 0.
I20240503 13:56:52.634481 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c5e25679b894ac4a86c2248c7f4f1bb" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
I20240503 13:56:52.634617 13447 raft_consensus.cc:2454] T 8c5e25679b894ac4a86c2248c7f4f1bb P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 0.
I20240503 13:56:52.634337 23911 tablet_bootstrap.cc:492] T 39b50d0b7d934e47ac61410e8173033b P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:52.635461 13118 leader_election.cc:302] T 8c5e25679b894ac4a86c2248c7f4f1bb P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, 8ccf5a4ac450414c9d3579bd5800b770; no voters:
I20240503 13:56:52.635896 23839 raft_consensus.cc:2781] T 8c5e25679b894ac4a86c2248c7f4f1bb P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:52.635972 23839 raft_consensus.cc:482] T 8c5e25679b894ac4a86c2248c7f4f1bb P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:52.636018 23839 raft_consensus.cc:3037] T 8c5e25679b894ac4a86c2248c7f4f1bb P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:52.636389 23911 tablet_bootstrap.cc:654] T 39b50d0b7d934e47ac61410e8173033b P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:52.637169 23839 raft_consensus.cc:504] T 8c5e25679b894ac4a86c2248c7f4f1bb P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:52.637421 23911 tablet_bootstrap.cc:492] T 39b50d0b7d934e47ac61410e8173033b P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:52.637539 23911 ts_tablet_manager.cc:1367] T 39b50d0b7d934e47ac61410e8173033b P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 13:56:52.637687 23839 leader_election.cc:288] T 8c5e25679b894ac4a86c2248c7f4f1bb P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 election: Requested vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951)
I20240503 13:56:52.637965 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c5e25679b894ac4a86c2248c7f4f1bb" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
I20240503 13:56:52.638026 15624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c5e25679b894ac4a86c2248c7f4f1bb" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d"
I20240503 13:56:52.638080 13445 raft_consensus.cc:3037] T 8c5e25679b894ac4a86c2248c7f4f1bb P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:52.638126 15624 raft_consensus.cc:3037] T 8c5e25679b894ac4a86c2248c7f4f1bb P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:52.638126 23911 raft_consensus.cc:348] T 39b50d0b7d934e47ac61410e8173033b P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:52.638214 23911 raft_consensus.cc:374] T 39b50d0b7d934e47ac61410e8173033b P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:52.638254 23911 raft_consensus.cc:729] T 39b50d0b7d934e47ac61410e8173033b P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:52.638375 23911 consensus_queue.cc:260] T 39b50d0b7d934e47ac61410e8173033b P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:52.638855 23909 tablet_bootstrap.cc:492] T 39b50d0b7d934e47ac61410e8173033b P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:52.638949 23909 ts_tablet_manager.cc:1367] T 39b50d0b7d934e47ac61410e8173033b P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 13:56:52.639114 13445 raft_consensus.cc:2454] T 8c5e25679b894ac4a86c2248c7f4f1bb P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 1.
I20240503 13:56:52.639132 15624 raft_consensus.cc:2454] T 8c5e25679b894ac4a86c2248c7f4f1bb P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 1.
I20240503 13:56:52.639386 23909 raft_consensus.cc:348] T 39b50d0b7d934e47ac61410e8173033b P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:52.639478 23909 raft_consensus.cc:374] T 39b50d0b7d934e47ac61410e8173033b P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:52.639504 23909 raft_consensus.cc:729] T 39b50d0b7d934e47ac61410e8173033b P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:52.639597 23909 consensus_queue.cc:260] T 39b50d0b7d934e47ac61410e8173033b P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:52.639850 23909 ts_tablet_manager.cc:1398] T 39b50d0b7d934e47ac61410e8173033b P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:52.639835 13118 leader_election.cc:302] T 8c5e25679b894ac4a86c2248c7f4f1bb P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, 8ccf5a4ac450414c9d3579bd5800b770; no voters:
I20240503 13:56:52.640029 23839 raft_consensus.cc:2781] T 8c5e25679b894ac4a86c2248c7f4f1bb P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:52.640161 23839 raft_consensus.cc:686] T 8c5e25679b894ac4a86c2248c7f4f1bb P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 LEADER]: Becoming Leader. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Running, Role: LEADER
I20240503 13:56:52.640290 23839 consensus_queue.cc:237] T 8c5e25679b894ac4a86c2248c7f4f1bb P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:52.640880 23911 ts_tablet_manager.cc:1398] T 39b50d0b7d934e47ac61410e8173033b P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20240503 13:56:52.642354 12907 catalog_manager.cc:5484] T 8c5e25679b894ac4a86c2248c7f4f1bb P 8ccf5a4ac450414c9d3579bd5800b770 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66). New cstate: current_term: 1 leader_uuid: "8ccf5a4ac450414c9d3579bd5800b770" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } }
I20240503 13:56:52.646551 23131 raft_consensus.cc:482] T 6c28bbb731834bf59af42d21a55fdffe P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:52.646651 23131 raft_consensus.cc:504] T 6c28bbb731834bf59af42d21a55fdffe P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:52.647065 23131 leader_election.cc:288] T 6c28bbb731834bf59af42d21a55fdffe P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279), 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951)
I20240503 13:56:52.647354 13316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c28bbb731834bf59af42d21a55fdffe" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703" is_pre_election: true
I20240503 13:56:52.647475 13316 raft_consensus.cc:2454] T 6c28bbb731834bf59af42d21a55fdffe P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 0.
I20240503 13:56:52.647501 15624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c28bbb731834bf59af42d21a55fdffe" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d" is_pre_election: true
I20240503 13:56:52.647642 15624 raft_consensus.cc:2454] T 6c28bbb731834bf59af42d21a55fdffe P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 0.
I20240503 13:56:52.647734 15694 leader_election.cc:302] T 6c28bbb731834bf59af42d21a55fdffe P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76c386bef61a4bd5a907266a5118ffe2, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:52.647881 23131 raft_consensus.cc:2781] T 6c28bbb731834bf59af42d21a55fdffe P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:52.647939 23131 raft_consensus.cc:482] T 6c28bbb731834bf59af42d21a55fdffe P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:52.647975 23131 raft_consensus.cc:3037] T 6c28bbb731834bf59af42d21a55fdffe P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:52.649127 23131 raft_consensus.cc:504] T 6c28bbb731834bf59af42d21a55fdffe P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:52.649472 23131 leader_election.cc:288] T 6c28bbb731834bf59af42d21a55fdffe P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 election: Requested vote from peers ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279), 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951)
I20240503 13:56:52.650383 13316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c28bbb731834bf59af42d21a55fdffe" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703"
I20240503 13:56:52.650382 15624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c28bbb731834bf59af42d21a55fdffe" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d"
I20240503 13:56:52.650490 15624 raft_consensus.cc:3037] T 6c28bbb731834bf59af42d21a55fdffe P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:52.650490 13316 raft_consensus.cc:3037] T 6c28bbb731834bf59af42d21a55fdffe P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:52.651401 13316 raft_consensus.cc:2454] T 6c28bbb731834bf59af42d21a55fdffe P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 1.
I20240503 13:56:52.651401 15624 raft_consensus.cc:2454] T 6c28bbb731834bf59af42d21a55fdffe P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 1.
I20240503 13:56:52.651834 15694 leader_election.cc:302] T 6c28bbb731834bf59af42d21a55fdffe P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76c386bef61a4bd5a907266a5118ffe2, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:52.652216 23131 raft_consensus.cc:2781] T 6c28bbb731834bf59af42d21a55fdffe P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:52.652333 23131 raft_consensus.cc:686] T 6c28bbb731834bf59af42d21a55fdffe P 76c386bef61a4bd5a907266a5118ffe2 [term 1 LEADER]: Becoming Leader. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Running, Role: LEADER
I20240503 13:56:52.652462 23131 consensus_queue.cc:237] T 6c28bbb731834bf59af42d21a55fdffe P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:52.654748 12906 catalog_manager.cc:5484] T 6c28bbb731834bf59af42d21a55fdffe P 76c386bef61a4bd5a907266a5118ffe2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71). New cstate: current_term: 1 leader_uuid: "76c386bef61a4bd5a907266a5118ffe2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:52.664665 23473 consensus_queue.cc:1035] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:52.667194 23397 consensus_queue.cc:1035] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:52.682529 23642 raft_consensus.cc:482] T 39b50d0b7d934e47ac61410e8173033b P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:52.682636 23642 raft_consensus.cc:504] T 39b50d0b7d934e47ac61410e8173033b P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:52.683063 23642 leader_election.cc:288] T 39b50d0b7d934e47ac61410e8173033b P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951), 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885)
I20240503 13:56:52.683413 15624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39b50d0b7d934e47ac61410e8173033b" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d" is_pre_election: true
I20240503 13:56:52.683548 15624 raft_consensus.cc:2454] T 39b50d0b7d934e47ac61410e8173033b P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 0.
I20240503 13:56:52.683549 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39b50d0b7d934e47ac61410e8173033b" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2" is_pre_election: true
I20240503 13:56:52.683677 15757 raft_consensus.cc:2454] T 39b50d0b7d934e47ac61410e8173033b P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 0.
I20240503 13:56:52.683908 15426 leader_election.cc:302] T 39b50d0b7d934e47ac61410e8173033b P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 61b887f0d40d497badeca5604c1f0546; no voters:
I20240503 13:56:52.684199 23642 raft_consensus.cc:2781] T 39b50d0b7d934e47ac61410e8173033b P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:52.684262 23642 raft_consensus.cc:482] T 39b50d0b7d934e47ac61410e8173033b P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:52.684307 23642 raft_consensus.cc:3037] T 39b50d0b7d934e47ac61410e8173033b P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:52.685273 23642 raft_consensus.cc:504] T 39b50d0b7d934e47ac61410e8173033b P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:52.685635 23642 leader_election.cc:288] T 39b50d0b7d934e47ac61410e8173033b P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 election: Requested vote from peers 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951), 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885)
I20240503 13:56:52.685937 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39b50d0b7d934e47ac61410e8173033b" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2"
I20240503 13:56:52.686002 15624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39b50d0b7d934e47ac61410e8173033b" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d"
I20240503 13:56:52.686084 15757 raft_consensus.cc:3037] T 39b50d0b7d934e47ac61410e8173033b P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:52.686100 15624 raft_consensus.cc:3037] T 39b50d0b7d934e47ac61410e8173033b P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:52.687057 15757 raft_consensus.cc:2454] T 39b50d0b7d934e47ac61410e8173033b P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 1.
I20240503 13:56:52.687130 15624 raft_consensus.cc:2454] T 39b50d0b7d934e47ac61410e8173033b P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 1.
I20240503 13:56:52.687379 15429 leader_election.cc:302] T 39b50d0b7d934e47ac61410e8173033b P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61b887f0d40d497badeca5604c1f0546, 76c386bef61a4bd5a907266a5118ffe2; no voters:
I20240503 13:56:52.687592 23642 raft_consensus.cc:2781] T 39b50d0b7d934e47ac61410e8173033b P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:52.687682 23642 raft_consensus.cc:686] T 39b50d0b7d934e47ac61410e8173033b P 61b887f0d40d497badeca5604c1f0546 [term 1 LEADER]: Becoming Leader. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Running, Role: LEADER
I20240503 13:56:52.687817 23642 consensus_queue.cc:237] T 39b50d0b7d934e47ac61410e8173033b P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:52.690232 12906 catalog_manager.cc:5484] T 39b50d0b7d934e47ac61410e8173033b P 61b887f0d40d497badeca5604c1f0546 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61b887f0d40d497badeca5604c1f0546 (127.6.98.69). New cstate: current_term: 1 leader_uuid: "61b887f0d40d497badeca5604c1f0546" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:52.738610 22256 raft_consensus.cc:482] T f64b8b4b4c4c4413b2da874c8751710e P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:52.738735 22256 raft_consensus.cc:504] T f64b8b4b4c4c4413b2da874c8751710e P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:52.739192 22256 leader_election.cc:288] T f64b8b4b4c4c4413b2da874c8751710e P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695), ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:52.739573 13316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f64b8b4b4c4c4413b2da874c8751710e" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703" is_pre_election: true
I20240503 13:56:52.739707 13316 raft_consensus.cc:2454] T f64b8b4b4c4c4413b2da874c8751710e P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 0.
I20240503 13:56:52.739969 13052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f64b8b4b4c4c4413b2da874c8751710e" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e" is_pre_election: true
I20240503 13:56:52.740093 13052 raft_consensus.cc:2454] T f64b8b4b4c4c4413b2da874c8751710e P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 0.
I20240503 13:56:52.740454 15823 leader_election.cc:302] T f64b8b4b4c4c4413b2da874c8751710e P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:52.740669 22256 raft_consensus.cc:2781] T f64b8b4b4c4c4413b2da874c8751710e P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:52.740715 22256 raft_consensus.cc:482] T f64b8b4b4c4c4413b2da874c8751710e P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:52.740748 22256 raft_consensus.cc:3037] T f64b8b4b4c4c4413b2da874c8751710e P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:52.741912 22256 raft_consensus.cc:504] T f64b8b4b4c4c4413b2da874c8751710e P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:52.742302 22256 leader_election.cc:288] T f64b8b4b4c4c4413b2da874c8751710e P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 election: Requested vote from peers 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695), ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:52.745853 13316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f64b8b4b4c4c4413b2da874c8751710e" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703"
I20240503 13:56:52.745993 13316 raft_consensus.cc:3037] T f64b8b4b4c4c4413b2da874c8751710e P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:52.747078 13316 raft_consensus.cc:2454] T f64b8b4b4c4c4413b2da874c8751710e P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 1.
I20240503 13:56:52.747594 15825 leader_election.cc:302] T f64b8b4b4c4c4413b2da874c8751710e P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba0194098aa84449ae17e8a0d4c43703, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:52.747797 22256 raft_consensus.cc:2781] T f64b8b4b4c4c4413b2da874c8751710e P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:52.747859 22256 raft_consensus.cc:686] T f64b8b4b4c4c4413b2da874c8751710e P e47105a0f46e4511b7112e571688a4c7 [term 1 LEADER]: Becoming Leader. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Running, Role: LEADER
I20240503 13:56:52.747992 22256 consensus_queue.cc:237] T f64b8b4b4c4c4413b2da874c8751710e P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:52.749131 13051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f64b8b4b4c4c4413b2da874c8751710e" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e"
I20240503 13:56:52.749243 13051 raft_consensus.cc:3037] T f64b8b4b4c4c4413b2da874c8751710e P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:52.750320 13051 raft_consensus.cc:2454] T f64b8b4b4c4c4413b2da874c8751710e P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 1.
I20240503 13:56:52.752295 12912 catalog_manager.cc:5484] T f64b8b4b4c4c4413b2da874c8751710e P e47105a0f46e4511b7112e571688a4c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e47105a0f46e4511b7112e571688a4c7 (127.6.98.72). New cstate: current_term: 1 leader_uuid: "e47105a0f46e4511b7112e571688a4c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:52.786715 15604 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 13:56:52.788736 13032 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 13:56:52.791618 13163 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:56:52.802971 13427 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:52.811275 15867 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 13:56:52.815919 15472 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 13:56:52.841229 15736 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 13:56:52.845798 13295 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 13:56:52.872680 23985 ts_tablet_manager.cc:902] T 7239f94a7c88433aa6062390a54e7562 P 009f849ee7b54c80a3f5264695047e4d: Initiating tablet copy from peer 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:52.873239 23985 tablet_copy_client.cc:308] T 7239f94a7c88433aa6062390a54e7562 P 009f849ee7b54c80a3f5264695047e4d: tablet copy: Beginning tablet copy session from remote peer at address 127.6.98.66:36387
I20240503 13:56:52.873800 13205 tablet_copy_service.cc:140] P 8ccf5a4ac450414c9d3579bd5800b770: Received BeginTabletCopySession request for tablet 7239f94a7c88433aa6062390a54e7562 from peer 009f849ee7b54c80a3f5264695047e4d ({username='slave'} at 127.6.98.70:57199)
I20240503 13:56:52.873906 13205 tablet_copy_service.cc:161] P 8ccf5a4ac450414c9d3579bd5800b770: Beginning new tablet copy session on tablet 7239f94a7c88433aa6062390a54e7562 from peer 009f849ee7b54c80a3f5264695047e4d at {username='slave'} at 127.6.98.70:57199: session id = 009f849ee7b54c80a3f5264695047e4d-7239f94a7c88433aa6062390a54e7562
I20240503 13:56:52.874917 13205 tablet_copy_source_session.cc:205] T 7239f94a7c88433aa6062390a54e7562 P 8ccf5a4ac450414c9d3579bd5800b770: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:56:52.875851 23985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7239f94a7c88433aa6062390a54e7562. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:52.878157 23985 tablet_copy_client.cc:783] T 7239f94a7c88433aa6062390a54e7562 P 009f849ee7b54c80a3f5264695047e4d: tablet copy: Starting download of 0 data blocks...
I20240503 13:56:52.878345 23985 tablet_copy_client.cc:647] T 7239f94a7c88433aa6062390a54e7562 P 009f849ee7b54c80a3f5264695047e4d: tablet copy: Starting download of 1 WAL segments...
I20240503 13:56:52.880311 23985 tablet_copy_client.cc:523] T 7239f94a7c88433aa6062390a54e7562 P 009f849ee7b54c80a3f5264695047e4d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:56:52.881779 23985 tablet_bootstrap.cc:492] T 7239f94a7c88433aa6062390a54e7562 P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:52.891872 12903 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:56:52.898018 13051 raft_consensus.cc:1261] T f64b8b4b4c4c4413b2da874c8751710e P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Refusing update from remote peer e47105a0f46e4511b7112e571688a4c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:52.898018 13314 raft_consensus.cc:1261] T f64b8b4b4c4c4413b2da874c8751710e P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Refusing update from remote peer e47105a0f46e4511b7112e571688a4c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:52.899092 22256 consensus_queue.cc:1035] T f64b8b4b4c4c4413b2da874c8751710e P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:52.899602 22256 consensus_queue.cc:1035] T f64b8b4b4c4c4413b2da874c8751710e P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:52.902115 12903 catalog_manager.cc:5576] TS 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695) does not have the latest schema for tablet f64b8b4b4c4c4413b2da874c8751710e (table rebalancer_dynamic_table [id=473ab6f00fab4abb9fb40247a66eb620]). Expected version 1 got 0
I20240503 13:56:52.902756 12906 catalog_manager.cc:5576] TS ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279) does not have the latest schema for tablet f64b8b4b4c4c4413b2da874c8751710e (table rebalancer_dynamic_table [id=473ab6f00fab4abb9fb40247a66eb620]). Expected version 1 got 0
I20240503 13:56:52.903050 24004 tablet.cc:1722] T f64b8b4b4c4c4413b2da874c8751710e P e47105a0f46e4511b7112e571688a4c7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:52.903551 24004 tablet.cc:1620] T f64b8b4b4c4c4413b2da874c8751710e P e47105a0f46e4511b7112e571688a4c7: MemRowSet was empty: no flush needed.
I20240503 13:56:52.904273 24005 tablet.cc:1722] T f64b8b4b4c4c4413b2da874c8751710e P 60ba3ffe3b7746469834cd91c84be14e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:52.904596 24005 tablet.cc:1620] T f64b8b4b4c4c4413b2da874c8751710e P 60ba3ffe3b7746469834cd91c84be14e: MemRowSet was empty: no flush needed.
I20240503 13:56:52.905457 12906 catalog_manager.cc:5576] TS 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695) does not have the latest schema for tablet f64b8b4b4c4c4413b2da874c8751710e (table rebalancer_dynamic_table [id=473ab6f00fab4abb9fb40247a66eb620]). Expected version 1 got 0
I20240503 13:56:52.906935 24011 tablet.cc:1722] T f64b8b4b4c4c4413b2da874c8751710e P ba0194098aa84449ae17e8a0d4c43703: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:52.907294 23985 tablet_bootstrap.cc:492] T 7239f94a7c88433aa6062390a54e7562 P 009f849ee7b54c80a3f5264695047e4d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:56:52.907377 24011 tablet.cc:1620] T f64b8b4b4c4c4413b2da874c8751710e P ba0194098aa84449ae17e8a0d4c43703: MemRowSet was empty: no flush needed.
I20240503 13:56:52.907622 23985 tablet_bootstrap.cc:492] T 7239f94a7c88433aa6062390a54e7562 P 009f849ee7b54c80a3f5264695047e4d: Bootstrap complete.
I20240503 13:56:52.907853 23985 ts_tablet_manager.cc:1367] T 7239f94a7c88433aa6062390a54e7562 P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.026s user 0.025s sys 0.002s
I20240503 13:56:52.908376 23985 raft_consensus.cc:348] T 7239f94a7c88433aa6062390a54e7562 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:52.908438 12906 catalog_manager.cc:5576] TS ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279) does not have the latest schema for tablet f64b8b4b4c4c4413b2da874c8751710e (table rebalancer_dynamic_table [id=473ab6f00fab4abb9fb40247a66eb620]). Expected version 1 got 0
I20240503 13:56:52.908634 23985 raft_consensus.cc:729] T 7239f94a7c88433aa6062390a54e7562 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:52.908818 23985 consensus_queue.cc:260] T 7239f94a7c88433aa6062390a54e7562 P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:52.909216 23985 ts_tablet_manager.cc:1398] T 7239f94a7c88433aa6062390a54e7562 P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:52.909812 13205 tablet_copy_service.cc:342] P 8ccf5a4ac450414c9d3579bd5800b770: Request end of tablet copy session 009f849ee7b54c80a3f5264695047e4d-7239f94a7c88433aa6062390a54e7562 received from {username='slave'} at 127.6.98.70:57199
I20240503 13:56:52.909893 13205 tablet_copy_service.cc:433] P 8ccf5a4ac450414c9d3579bd5800b770: ending tablet copy session 009f849ee7b54c80a3f5264695047e4d-7239f94a7c88433aa6062390a54e7562 on tablet 7239f94a7c88433aa6062390a54e7562 with peer 009f849ee7b54c80a3f5264695047e4d
I20240503 13:56:52.916388 12906 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=473ab6f00fab4abb9fb40247a66eb620] alter complete (version 1)
I20240503 13:56:52.921365 13314 consensus_queue.cc:237] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:52.922735 15888 raft_consensus.cc:1261] T 84f1ab11e74045f8a9d7033c5ead0c14 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:56:52.923126 13182 raft_consensus.cc:1261] T 84f1ab11e74045f8a9d7033c5ead0c14 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:56:52.923197 23397 consensus_queue.cc:1035] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:52.923736 23397 consensus_queue.cc:1035] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:52.925218 23473 raft_consensus.cc:2932] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:52.925912 13182 raft_consensus.cc:2932] T 84f1ab11e74045f8a9d7033c5ead0c14 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:52.928169 15888 raft_consensus.cc:2932] T 84f1ab11e74045f8a9d7033c5ead0c14 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:52.928458 12906 catalog_manager.cc:5484] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 reported cstate change: config changed from index 4 to 5, VOTER 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68) evicted. New cstate: current_term: 1 leader_uuid: "ba0194098aa84449ae17e8a0d4c43703" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:52.932525 13427 tablet_service.cc:1514] Processing DeleteTablet for tablet 84f1ab11e74045f8a9d7033c5ead0c14 with delete_type TABLET_DATA_TOMBSTONED (TS 7690ee79577c45e3a2c8f8b552c0bff8 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:33678
I20240503 13:56:52.932879 24017 tablet_replica.cc:330] T 84f1ab11e74045f8a9d7033c5ead0c14 P 7690ee79577c45e3a2c8f8b552c0bff8: stopping tablet replica
I20240503 13:56:52.933025 24017 raft_consensus.cc:2229] T 84f1ab11e74045f8a9d7033c5ead0c14 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:52.933384 24017 raft_consensus.cc:2258] T 84f1ab11e74045f8a9d7033c5ead0c14 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:52.934015 24017 ts_tablet_manager.cc:1875] T 84f1ab11e74045f8a9d7033c5ead0c14 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:56:52.936462 24017 ts_tablet_manager.cc:1888] T 84f1ab11e74045f8a9d7033c5ead0c14 P 7690ee79577c45e3a2c8f8b552c0bff8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 13:56:52.936578 24017 log.cc:1196] T 84f1ab11e74045f8a9d7033c5ead0c14 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal/wals/84f1ab11e74045f8a9d7033c5ead0c14
I20240503 13:56:52.937129 12896 catalog_manager.cc:4830] TS 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109): tablet 84f1ab11e74045f8a9d7033c5ead0c14 (table rebalance_test_table_19 [id=e6609c16048348029c01374cb5a68ce8]) successfully deleted
I20240503 13:56:53.022087 12907 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:56:53.030687 24004 tablet.cc:1722] T f64b8b4b4c4c4413b2da874c8751710e P e47105a0f46e4511b7112e571688a4c7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:53.031041 24004 tablet.cc:1620] T f64b8b4b4c4c4413b2da874c8751710e P e47105a0f46e4511b7112e571688a4c7: MemRowSet was empty: no flush needed.
I20240503 13:56:53.031226 24011 tablet.cc:1722] T f64b8b4b4c4c4413b2da874c8751710e P ba0194098aa84449ae17e8a0d4c43703: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:53.031515 24011 tablet.cc:1620] T f64b8b4b4c4c4413b2da874c8751710e P ba0194098aa84449ae17e8a0d4c43703: MemRowSet was empty: no flush needed.
I20240503 13:56:53.036679 24005 tablet.cc:1722] T f64b8b4b4c4c4413b2da874c8751710e P 60ba3ffe3b7746469834cd91c84be14e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:53.037006 24005 tablet.cc:1620] T f64b8b4b4c4c4413b2da874c8751710e P 60ba3ffe3b7746469834cd91c84be14e: MemRowSet was empty: no flush needed.
I20240503 13:56:53.039260 12907 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=473ab6f00fab4abb9fb40247a66eb620] alter complete (version 2)
I20240503 13:56:53.050262 23843 consensus_queue.cc:1035] T 8c5e25679b894ac4a86c2248c7f4f1bb P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:53.054519 23843 consensus_queue.cc:1035] T 8c5e25679b894ac4a86c2248c7f4f1bb P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:53.069916 15604 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 13:56:53.073460 15472 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 13:56:53.080169 13032 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 13:56:53.087077 13163 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:56:53.088636 15736 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 13:56:53.094180 24013 consensus_queue.cc:1035] T 6c28bbb731834bf59af42d21a55fdffe P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:53.094653 13295 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 13:56:53.102813 23094 consensus_queue.cc:1035] T 39b50d0b7d934e47ac61410e8173033b P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:53.111572 23131 consensus_queue.cc:1035] T 6c28bbb731834bf59af42d21a55fdffe P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:53.123824 13427 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:53.130785 15867 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 13:56:53.136575 23094 consensus_queue.cc:1035] T 39b50d0b7d934e47ac61410e8173033b P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:53.148100 12912 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:56:53.154529 24004 tablet.cc:1722] T f64b8b4b4c4c4413b2da874c8751710e P e47105a0f46e4511b7112e571688a4c7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:53.157156 24011 tablet.cc:1722] T f64b8b4b4c4c4413b2da874c8751710e P ba0194098aa84449ae17e8a0d4c43703: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:53.160032 24005 tablet.cc:1722] T f64b8b4b4c4c4413b2da874c8751710e P 60ba3ffe3b7746469834cd91c84be14e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:53.165743 12912 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table_new_name [id=473ab6f00fab4abb9fb40247a66eb620] alter complete (version 3)
I20240503 13:56:53.193902 12907 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_extra_table_62"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:56:53.201809 15867 tablet_service.cc:1467] Processing CreateTablet for tablet 3da6bc0d649643e1acd8d8811ad1f5ad (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=388a8a04fd3d43cc981b6940d8244a5a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:53.201869 15736 tablet_service.cc:1467] Processing CreateTablet for tablet 3da6bc0d649643e1acd8d8811ad1f5ad (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=388a8a04fd3d43cc981b6940d8244a5a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:53.202128 15736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3da6bc0d649643e1acd8d8811ad1f5ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:53.202129 15867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3da6bc0d649643e1acd8d8811ad1f5ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:53.202626 13163 tablet_service.cc:1467] Processing CreateTablet for tablet 3da6bc0d649643e1acd8d8811ad1f5ad (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=388a8a04fd3d43cc981b6940d8244a5a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:53.202883 13163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3da6bc0d649643e1acd8d8811ad1f5ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:53.205185 13034 tablet_service.cc:1467] Processing CreateTablet for tablet 5977344952664d0c8165dcd1f3cea068 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=388a8a04fd3d43cc981b6940d8244a5a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:53.205278 13427 tablet_service.cc:1467] Processing CreateTablet for tablet fc7c89676b1b4262a7579483ccfe8991 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=388a8a04fd3d43cc981b6940d8244a5a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:53.205485 13034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5977344952664d0c8165dcd1f3cea068. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:53.205557 13427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc7c89676b1b4262a7579483ccfe8991. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:53.205765 13164 tablet_service.cc:1467] Processing CreateTablet for tablet 5977344952664d0c8165dcd1f3cea068 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=388a8a04fd3d43cc981b6940d8244a5a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:53.205986 13164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5977344952664d0c8165dcd1f3cea068. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:53.207337 24084 tablet_bootstrap.cc:492] T 5977344952664d0c8165dcd1f3cea068 P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:53.205185 13032 tablet_service.cc:1467] Processing CreateTablet for tablet fc7c89676b1b4262a7579483ccfe8991 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=388a8a04fd3d43cc981b6940d8244a5a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:53.207607 13032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc7c89676b1b4262a7579483ccfe8991. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:53.208700 24084 tablet_bootstrap.cc:654] T 5977344952664d0c8165dcd1f3cea068 P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:53.209707 24084 tablet_bootstrap.cc:492] T 5977344952664d0c8165dcd1f3cea068 P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:53.209784 24084 ts_tablet_manager.cc:1367] T 5977344952664d0c8165dcd1f3cea068 P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 13:56:53.210211 24084 raft_consensus.cc:348] T 5977344952664d0c8165dcd1f3cea068 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:53.210306 24084 raft_consensus.cc:374] T 5977344952664d0c8165dcd1f3cea068 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:53.210336 24084 raft_consensus.cc:729] T 5977344952664d0c8165dcd1f3cea068 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:53.210455 24086 tablet_bootstrap.cc:492] T 5977344952664d0c8165dcd1f3cea068 P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:53.210443 24084 consensus_queue.cc:260] T 5977344952664d0c8165dcd1f3cea068 P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:53.210747 24084 ts_tablet_manager.cc:1398] T 5977344952664d0c8165dcd1f3cea068 P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:53.210922 24084 tablet_bootstrap.cc:492] T fc7c89676b1b4262a7579483ccfe8991 P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:53.211315 24086 tablet_bootstrap.cc:654] T 5977344952664d0c8165dcd1f3cea068 P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:53.211738 24084 tablet_bootstrap.cc:654] T fc7c89676b1b4262a7579483ccfe8991 P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:53.211918 15472 tablet_service.cc:1467] Processing CreateTablet for tablet fc7c89676b1b4262a7579483ccfe8991 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=388a8a04fd3d43cc981b6940d8244a5a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:53.212149 15472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc7c89676b1b4262a7579483ccfe8991. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:53.212174 15473 tablet_service.cc:1467] Processing CreateTablet for tablet 5977344952664d0c8165dcd1f3cea068 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=388a8a04fd3d43cc981b6940d8244a5a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:53.212430 15473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5977344952664d0c8165dcd1f3cea068. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:53.214082 24082 tablet_bootstrap.cc:492] T 3da6bc0d649643e1acd8d8811ad1f5ad P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:53.214953 24082 tablet_bootstrap.cc:654] T 3da6bc0d649643e1acd8d8811ad1f5ad P e47105a0f46e4511b7112e571688a4c7: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:53.215667 24083 tablet_bootstrap.cc:492] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:53.217782 24084 tablet_bootstrap.cc:492] T fc7c89676b1b4262a7579483ccfe8991 P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:53.217888 24084 ts_tablet_manager.cc:1367] T fc7c89676b1b4262a7579483ccfe8991 P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240503 13:56:53.218036 24085 tablet_bootstrap.cc:492] T fc7c89676b1b4262a7579483ccfe8991 P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:53.218344 24084 raft_consensus.cc:348] T fc7c89676b1b4262a7579483ccfe8991 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:53.218454 24084 raft_consensus.cc:374] T fc7c89676b1b4262a7579483ccfe8991 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:53.218483 24084 raft_consensus.cc:729] T fc7c89676b1b4262a7579483ccfe8991 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:53.218611 24084 consensus_queue.cc:260] T fc7c89676b1b4262a7579483ccfe8991 P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:53.218981 24085 tablet_bootstrap.cc:654] T fc7c89676b1b4262a7579483ccfe8991 P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:53.221189 24086 tablet_bootstrap.cc:492] T 5977344952664d0c8165dcd1f3cea068 P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:53.221374 24086 ts_tablet_manager.cc:1367] T 5977344952664d0c8165dcd1f3cea068 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 13:56:53.221652 24083 tablet_bootstrap.cc:654] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:53.221865 24084 ts_tablet_manager.cc:1398] T fc7c89676b1b4262a7579483ccfe8991 P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240503 13:56:53.222119 24086 raft_consensus.cc:348] T 5977344952664d0c8165dcd1f3cea068 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:53.222234 24086 raft_consensus.cc:374] T 5977344952664d0c8165dcd1f3cea068 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:53.222265 24086 raft_consensus.cc:729] T 5977344952664d0c8165dcd1f3cea068 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:53.222424 24086 consensus_queue.cc:260] T 5977344952664d0c8165dcd1f3cea068 P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:53.222522 24090 tablet_bootstrap.cc:492] T fc7c89676b1b4262a7579483ccfe8991 P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:53.222934 24086 ts_tablet_manager.cc:1398] T 5977344952664d0c8165dcd1f3cea068 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:53.223313 24086 tablet_bootstrap.cc:492] T 3da6bc0d649643e1acd8d8811ad1f5ad P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:53.224406 24090 tablet_bootstrap.cc:654] T fc7c89676b1b4262a7579483ccfe8991 P 61b887f0d40d497badeca5604c1f0546: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:53.224577 24086 tablet_bootstrap.cc:654] T 3da6bc0d649643e1acd8d8811ad1f5ad P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:53.224759 24085 tablet_bootstrap.cc:492] T fc7c89676b1b4262a7579483ccfe8991 P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:53.225481 24085 ts_tablet_manager.cc:1367] T fc7c89676b1b4262a7579483ccfe8991 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 13:56:53.226570 24082 tablet_bootstrap.cc:492] T 3da6bc0d649643e1acd8d8811ad1f5ad P e47105a0f46e4511b7112e571688a4c7: No bootstrap required, opened a new log
I20240503 13:56:53.226722 24082 ts_tablet_manager.cc:1367] T 3da6bc0d649643e1acd8d8811ad1f5ad P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 13:56:53.227252 24082 raft_consensus.cc:348] T 3da6bc0d649643e1acd8d8811ad1f5ad P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:53.227439 24082 raft_consensus.cc:374] T 3da6bc0d649643e1acd8d8811ad1f5ad P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:53.227525 24082 raft_consensus.cc:729] T 3da6bc0d649643e1acd8d8811ad1f5ad P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:53.227732 24085 raft_consensus.cc:348] T fc7c89676b1b4262a7579483ccfe8991 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:53.227823 24082 consensus_queue.cc:260] T 3da6bc0d649643e1acd8d8811ad1f5ad P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:53.227902 24085 raft_consensus.cc:374] T fc7c89676b1b4262a7579483ccfe8991 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:53.227934 24085 raft_consensus.cc:729] T fc7c89676b1b4262a7579483ccfe8991 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:53.228093 24085 consensus_queue.cc:260] T fc7c89676b1b4262a7579483ccfe8991 P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:53.228431 24085 ts_tablet_manager.cc:1398] T fc7c89676b1b4262a7579483ccfe8991 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:53.229048 24082 ts_tablet_manager.cc:1398] T 3da6bc0d649643e1acd8d8811ad1f5ad P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:53.229992 24083 tablet_bootstrap.cc:492] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:53.230154 24083 ts_tablet_manager.cc:1367] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240503 13:56:53.230646 24090 tablet_bootstrap.cc:492] T fc7c89676b1b4262a7579483ccfe8991 P 61b887f0d40d497badeca5604c1f0546: No bootstrap required, opened a new log
I20240503 13:56:53.230765 24090 ts_tablet_manager.cc:1367] T fc7c89676b1b4262a7579483ccfe8991 P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 13:56:53.231210 24090 raft_consensus.cc:348] T fc7c89676b1b4262a7579483ccfe8991 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:53.231657 24083 raft_consensus.cc:348] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:53.231822 24083 raft_consensus.cc:374] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:53.231895 24083 raft_consensus.cc:729] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:53.232064 24083 consensus_queue.cc:260] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:53.232481 24083 ts_tablet_manager.cc:1398] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:53.232846 24090 raft_consensus.cc:374] T fc7c89676b1b4262a7579483ccfe8991 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:53.232932 24090 raft_consensus.cc:729] T fc7c89676b1b4262a7579483ccfe8991 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:53.233134 24090 consensus_queue.cc:260] T fc7c89676b1b4262a7579483ccfe8991 P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:53.233480 24090 ts_tablet_manager.cc:1398] T fc7c89676b1b4262a7579483ccfe8991 P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:56:53.233579 24086 tablet_bootstrap.cc:492] T 3da6bc0d649643e1acd8d8811ad1f5ad P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:53.233712 24086 ts_tablet_manager.cc:1367] T 3da6bc0d649643e1acd8d8811ad1f5ad P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.001s
I20240503 13:56:53.233776 24090 tablet_bootstrap.cc:492] T 5977344952664d0c8165dcd1f3cea068 P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:53.234194 24086 raft_consensus.cc:348] T 3da6bc0d649643e1acd8d8811ad1f5ad P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:53.234354 24086 raft_consensus.cc:374] T 3da6bc0d649643e1acd8d8811ad1f5ad P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:53.234452 24086 raft_consensus.cc:729] T 3da6bc0d649643e1acd8d8811ad1f5ad P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:53.234658 24086 consensus_queue.cc:260] T 3da6bc0d649643e1acd8d8811ad1f5ad P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:53.235705 24086 ts_tablet_manager.cc:1398] T 3da6bc0d649643e1acd8d8811ad1f5ad P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:56:53.237237 24090 tablet_bootstrap.cc:654] T 5977344952664d0c8165dcd1f3cea068 P 61b887f0d40d497badeca5604c1f0546: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:53.238472 24090 tablet_bootstrap.cc:492] T 5977344952664d0c8165dcd1f3cea068 P 61b887f0d40d497badeca5604c1f0546: No bootstrap required, opened a new log
I20240503 13:56:53.238602 24090 ts_tablet_manager.cc:1367] T 5977344952664d0c8165dcd1f3cea068 P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:56:53.240411 24090 raft_consensus.cc:348] T 5977344952664d0c8165dcd1f3cea068 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:53.240576 24090 raft_consensus.cc:374] T 5977344952664d0c8165dcd1f3cea068 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:53.240636 24090 raft_consensus.cc:729] T 5977344952664d0c8165dcd1f3cea068 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:53.240780 24090 consensus_queue.cc:260] T 5977344952664d0c8165dcd1f3cea068 P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:53.243463 24090 ts_tablet_manager.cc:1398] T 5977344952664d0c8165dcd1f3cea068 P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20240503 13:56:53.269472 12904 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:53.269620 12904 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:53.271224 15888 consensus_queue.cc:237] T 3a140864f59b483bb5fea3de83d28ecf P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:53.271787 12904 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Sending DeleteTablet for 3 replicas of tablet f64b8b4b4c4c4413b2da874c8751710e
I20240503 13:56:53.272542 15623 raft_consensus.cc:1261] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Refusing update from remote peer e47105a0f46e4511b7112e571688a4c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:56:53.272548 13316 raft_consensus.cc:1261] T 3a140864f59b483bb5fea3de83d28ecf P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Refusing update from remote peer e47105a0f46e4511b7112e571688a4c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 13:56:53.272791 13032 tablet_service.cc:1514] Processing DeleteTablet for tablet f64b8b4b4c4c4413b2da874c8751710e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:53 UTC) from {username='slave'} at 127.0.0.1:59458
I20240503 13:56:53.272987 23617 consensus_queue.cc:1035] T 3a140864f59b483bb5fea3de83d28ecf P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:56:53.273037 15867 tablet_service.cc:1514] Processing DeleteTablet for tablet f64b8b4b4c4c4413b2da874c8751710e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:53 UTC) from {username='slave'} at 127.0.0.1:50498
I20240503 13:56:53.273241 13295 tablet_service.cc:1514] Processing DeleteTablet for tablet f64b8b4b4c4c4413b2da874c8751710e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:53 UTC) from {username='slave'} at 127.0.0.1:33250
I20240503 13:56:53.273592 23617 consensus_queue.cc:1035] T 3a140864f59b483bb5fea3de83d28ecf P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:56:53.274447 24114 tablet_replica.cc:330] T f64b8b4b4c4c4413b2da874c8751710e P 60ba3ffe3b7746469834cd91c84be14e: stopping tablet replica
I20240503 13:56:53.274647 24114 raft_consensus.cc:2229] T f64b8b4b4c4c4413b2da874c8751710e P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:53.274916 24114 raft_consensus.cc:2258] T f64b8b4b4c4c4413b2da874c8751710e P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:53.275832 24114 ts_tablet_manager.cc:1875] T f64b8b4b4c4c4413b2da874c8751710e P 60ba3ffe3b7746469834cd91c84be14e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:53.276100 24115 tablet_replica.cc:330] T f64b8b4b4c4c4413b2da874c8751710e P e47105a0f46e4511b7112e571688a4c7: stopping tablet replica
I20240503 13:56:53.276244 24115 raft_consensus.cc:2229] T f64b8b4b4c4c4413b2da874c8751710e P e47105a0f46e4511b7112e571688a4c7 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:56:53.276414 24115 raft_consensus.cc:2258] T f64b8b4b4c4c4413b2da874c8751710e P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:53.277185 24115 ts_tablet_manager.cc:1875] T f64b8b4b4c4c4413b2da874c8751710e P e47105a0f46e4511b7112e571688a4c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:53.278077 24114 ts_tablet_manager.cc:1888] T f64b8b4b4c4c4413b2da874c8751710e P 60ba3ffe3b7746469834cd91c84be14e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:53.278142 24114 log.cc:1196] T f64b8b4b4c4c4413b2da874c8751710e P 60ba3ffe3b7746469834cd91c84be14e: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal/wals/f64b8b4b4c4c4413b2da874c8751710e
I20240503 13:56:53.278616 24116 tablet_replica.cc:330] T f64b8b4b4c4c4413b2da874c8751710e P ba0194098aa84449ae17e8a0d4c43703: stopping tablet replica
I20240503 13:56:53.278630 24114 ts_tablet_manager.cc:1909] T f64b8b4b4c4c4413b2da874c8751710e P 60ba3ffe3b7746469834cd91c84be14e: Deleting consensus metadata
I20240503 13:56:53.278717 23617 raft_consensus.cc:2932] T 3a140864f59b483bb5fea3de83d28ecf P e47105a0f46e4511b7112e571688a4c7 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:53.279273 24116 raft_consensus.cc:2229] T f64b8b4b4c4c4413b2da874c8751710e P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:53.279387 24116 raft_consensus.cc:2258] T f64b8b4b4c4c4413b2da874c8751710e P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:53.280074 24116 ts_tablet_manager.cc:1875] T f64b8b4b4c4c4413b2da874c8751710e P ba0194098aa84449ae17e8a0d4c43703: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:53.280325 24115 ts_tablet_manager.cc:1888] T f64b8b4b4c4c4413b2da874c8751710e P e47105a0f46e4511b7112e571688a4c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:53.280392 24115 log.cc:1196] T f64b8b4b4c4c4413b2da874c8751710e P e47105a0f46e4511b7112e571688a4c7: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-7/wal/wals/f64b8b4b4c4c4413b2da874c8751710e
I20240503 13:56:53.281143 12897 catalog_manager.cc:4830] TS 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695): tablet f64b8b4b4c4c4413b2da874c8751710e (table rebalancer_dynamic_table_new_name [id=473ab6f00fab4abb9fb40247a66eb620]) successfully deleted
I20240503 13:56:53.281136 13316 raft_consensus.cc:2932] T 3a140864f59b483bb5fea3de83d28ecf P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:53.282464 24116 ts_tablet_manager.cc:1888] T f64b8b4b4c4c4413b2da874c8751710e P ba0194098aa84449ae17e8a0d4c43703: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:53.282579 24116 log.cc:1196] T f64b8b4b4c4c4413b2da874c8751710e P ba0194098aa84449ae17e8a0d4c43703: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/wal/wals/f64b8b4b4c4c4413b2da874c8751710e
I20240503 13:56:53.282657 12904 catalog_manager.cc:5484] T 3a140864f59b483bb5fea3de83d28ecf P e47105a0f46e4511b7112e571688a4c7 reported cstate change: config changed from index 2 to 3, VOTER 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71) evicted. New cstate: current_term: 1 leader_uuid: "e47105a0f46e4511b7112e571688a4c7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:53.282696 15623 raft_consensus.cc:2932] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:53.282953 24116 ts_tablet_manager.cc:1909] T f64b8b4b4c4c4413b2da874c8751710e P ba0194098aa84449ae17e8a0d4c43703: Deleting consensus metadata
I20240503 13:56:53.283748 12899 catalog_manager.cc:4830] TS ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279): tablet f64b8b4b4c4c4413b2da874c8751710e (table rebalancer_dynamic_table_new_name [id=473ab6f00fab4abb9fb40247a66eb620]) successfully deleted
I20240503 13:56:53.283954 24115 ts_tablet_manager.cc:1909] T f64b8b4b4c4c4413b2da874c8751710e P e47105a0f46e4511b7112e571688a4c7: Deleting consensus metadata
I20240503 13:56:53.288205 12896 catalog_manager.cc:4830] TS e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): tablet f64b8b4b4c4c4413b2da874c8751710e (table rebalancer_dynamic_table_new_name [id=473ab6f00fab4abb9fb40247a66eb620]) successfully deleted
I20240503 13:56:53.291532 15736 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a140864f59b483bb5fea3de83d28ecf with delete_type TABLET_DATA_TOMBSTONED (TS 76c386bef61a4bd5a907266a5118ffe2 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:44984
I20240503 13:56:53.292055 24124 tablet_replica.cc:330] T 3a140864f59b483bb5fea3de83d28ecf P 76c386bef61a4bd5a907266a5118ffe2: stopping tablet replica
I20240503 13:56:53.292225 24124 raft_consensus.cc:2229] T 3a140864f59b483bb5fea3de83d28ecf P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:53.292346 24124 raft_consensus.cc:2258] T 3a140864f59b483bb5fea3de83d28ecf P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:53.293248 23131 raft_consensus.cc:482] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:53.293326 23131 raft_consensus.cc:504] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:53.293668 23131 leader_election.cc:288] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:53.293932 13182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3da6bc0d649643e1acd8d8811ad1f5ad" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770" is_pre_election: true
I20240503 13:56:53.294054 15888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3da6bc0d649643e1acd8d8811ad1f5ad" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7" is_pre_election: true
I20240503 13:56:53.294107 13182 raft_consensus.cc:2454] T 3da6bc0d649643e1acd8d8811ad1f5ad P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 0.
I20240503 13:56:53.294162 15888 raft_consensus.cc:2454] T 3da6bc0d649643e1acd8d8811ad1f5ad P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 0.
I20240503 13:56:53.294373 15694 leader_election.cc:302] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76c386bef61a4bd5a907266a5118ffe2, 8ccf5a4ac450414c9d3579bd5800b770; no voters:
I20240503 13:56:53.293160 24124 ts_tablet_manager.cc:1875] T 3a140864f59b483bb5fea3de83d28ecf P 76c386bef61a4bd5a907266a5118ffe2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:56:53.294678 23686 raft_consensus.cc:2781] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:53.294731 23686 raft_consensus.cc:482] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:53.294764 23686 raft_consensus.cc:3037] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:53.295704 23686 raft_consensus.cc:504] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:53.296028 23686 leader_election.cc:288] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 election: Requested vote from peers 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:53.296350 15888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3da6bc0d649643e1acd8d8811ad1f5ad" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7"
I20240503 13:56:53.296350 13182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3da6bc0d649643e1acd8d8811ad1f5ad" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770"
I20240503 13:56:53.296470 15888 raft_consensus.cc:3037] T 3da6bc0d649643e1acd8d8811ad1f5ad P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:53.296466 13182 raft_consensus.cc:3037] T 3da6bc0d649643e1acd8d8811ad1f5ad P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:53.296698 24124 ts_tablet_manager.cc:1888] T 3a140864f59b483bb5fea3de83d28ecf P 76c386bef61a4bd5a907266a5118ffe2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 13:56:53.296746 24124 log.cc:1196] T 3a140864f59b483bb5fea3de83d28ecf P 76c386bef61a4bd5a907266a5118ffe2: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/wal/wals/3a140864f59b483bb5fea3de83d28ecf
I20240503 13:56:53.297415 13182 raft_consensus.cc:2454] T 3da6bc0d649643e1acd8d8811ad1f5ad P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 1.
I20240503 13:56:53.297405 15888 raft_consensus.cc:2454] T 3da6bc0d649643e1acd8d8811ad1f5ad P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 1.
I20240503 13:56:53.297768 15694 leader_election.cc:302] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76c386bef61a4bd5a907266a5118ffe2, 8ccf5a4ac450414c9d3579bd5800b770; no voters:
I20240503 13:56:53.297966 23686 raft_consensus.cc:2781] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:53.297991 12899 catalog_manager.cc:4830] TS 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885): tablet 3a140864f59b483bb5fea3de83d28ecf (table rebalancer_extra_table_4 [id=51ad326dfca444208bdd2a00d9582876]) successfully deleted
I20240503 13:56:53.298051 23686 raft_consensus.cc:686] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 [term 1 LEADER]: Becoming Leader. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Running, Role: LEADER
I20240503 13:56:53.298171 23686 consensus_queue.cc:237] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:53.300504 12907 catalog_manager.cc:5484] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71). New cstate: current_term: 1 leader_uuid: "76c386bef61a4bd5a907266a5118ffe2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:53.334121 15623 raft_consensus.cc:1203] T 7239f94a7c88433aa6062390a54e7562 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:56:53.371627 23094 raft_consensus.cc:482] T fc7c89676b1b4262a7579483ccfe8991 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:53.371745 23094 raft_consensus.cc:504] T fc7c89676b1b4262a7579483ccfe8991 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:53.372210 23094 leader_election.cc:288] T fc7c89676b1b4262a7579483ccfe8991 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:53.372556 13051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc7c89676b1b4262a7579483ccfe8991" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e" is_pre_election: true
I20240503 13:56:53.372705 13051 raft_consensus.cc:2454] T fc7c89676b1b4262a7579483ccfe8991 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 0.
I20240503 13:56:53.373126 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc7c89676b1b4262a7579483ccfe8991" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
I20240503 13:56:53.373261 13445 raft_consensus.cc:2454] T fc7c89676b1b4262a7579483ccfe8991 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 0.
I20240503 13:56:53.373800 15426 leader_election.cc:302] T fc7c89676b1b4262a7579483ccfe8991 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61b887f0d40d497badeca5604c1f0546, 7690ee79577c45e3a2c8f8b552c0bff8; no voters:
I20240503 13:56:53.373966 23094 raft_consensus.cc:2781] T fc7c89676b1b4262a7579483ccfe8991 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:53.374044 23094 raft_consensus.cc:482] T fc7c89676b1b4262a7579483ccfe8991 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:53.374096 23094 raft_consensus.cc:3037] T fc7c89676b1b4262a7579483ccfe8991 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:53.374487 20870 raft_consensus.cc:482] T 5977344952664d0c8165dcd1f3cea068 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:53.374588 20870 raft_consensus.cc:504] T 5977344952664d0c8165dcd1f3cea068 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:53.374946 20870 leader_election.cc:288] T 5977344952664d0c8165dcd1f3cea068 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387), 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153)
I20240503 13:56:53.375147 23094 raft_consensus.cc:504] T fc7c89676b1b4262a7579483ccfe8991 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:53.375432 15492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5977344952664d0c8165dcd1f3cea068" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546" is_pre_election: true
I20240503 13:56:53.375430 13182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5977344952664d0c8165dcd1f3cea068" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770" is_pre_election: true
I20240503 13:56:53.375486 23094 leader_election.cc:288] T fc7c89676b1b4262a7579483ccfe8991 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 election: Requested vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:53.375538 13182 raft_consensus.cc:2454] T 5977344952664d0c8165dcd1f3cea068 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 0.
I20240503 13:56:53.375548 15492 raft_consensus.cc:2454] T 5977344952664d0c8165dcd1f3cea068 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 0.
I20240503 13:56:53.375828 12990 leader_election.cc:302] T 5977344952664d0c8165dcd1f3cea068 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 8ccf5a4ac450414c9d3579bd5800b770; no voters:
I20240503 13:56:53.376226 13051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc7c89676b1b4262a7579483ccfe8991" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e"
I20240503 13:56:53.376329 13051 raft_consensus.cc:3037] T fc7c89676b1b4262a7579483ccfe8991 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:53.377060 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc7c89676b1b4262a7579483ccfe8991" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
I20240503 13:56:53.377172 13445 raft_consensus.cc:3037] T fc7c89676b1b4262a7579483ccfe8991 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:53.377441 13051 raft_consensus.cc:2454] T fc7c89676b1b4262a7579483ccfe8991 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 1.
I20240503 13:56:53.377775 15427 leader_election.cc:302] T fc7c89676b1b4262a7579483ccfe8991 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 61b887f0d40d497badeca5604c1f0546; no voters:
I20240503 13:56:53.377933 23094 raft_consensus.cc:2781] T fc7c89676b1b4262a7579483ccfe8991 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:53.378003 23094 raft_consensus.cc:686] T fc7c89676b1b4262a7579483ccfe8991 P 61b887f0d40d497badeca5604c1f0546 [term 1 LEADER]: Becoming Leader. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Running, Role: LEADER
I20240503 13:56:53.378149 23094 consensus_queue.cc:237] T fc7c89676b1b4262a7579483ccfe8991 P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:53.378201 13445 raft_consensus.cc:2454] T fc7c89676b1b4262a7579483ccfe8991 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 1.
I20240503 13:56:53.378867 22738 raft_consensus.cc:2781] T 5977344952664d0c8165dcd1f3cea068 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:53.378919 22738 raft_consensus.cc:482] T 5977344952664d0c8165dcd1f3cea068 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:53.378955 22738 raft_consensus.cc:3037] T 5977344952664d0c8165dcd1f3cea068 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:53.379791 22738 raft_consensus.cc:504] T 5977344952664d0c8165dcd1f3cea068 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:53.380147 22738 leader_election.cc:288] T 5977344952664d0c8165dcd1f3cea068 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 election: Requested vote from peers 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387), 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153)
I20240503 13:56:53.380949 13182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5977344952664d0c8165dcd1f3cea068" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770"
I20240503 13:56:53.381074 13182 raft_consensus.cc:3037] T 5977344952664d0c8165dcd1f3cea068 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:53.381191 12911 catalog_manager.cc:5484] T fc7c89676b1b4262a7579483ccfe8991 P 61b887f0d40d497badeca5604c1f0546 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61b887f0d40d497badeca5604c1f0546 (127.6.98.69). New cstate: current_term: 1 leader_uuid: "61b887f0d40d497badeca5604c1f0546" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } }
I20240503 13:56:53.381851 13182 raft_consensus.cc:2454] T 5977344952664d0c8165dcd1f3cea068 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 1.
I20240503 13:56:53.382136 12990 leader_election.cc:302] T 5977344952664d0c8165dcd1f3cea068 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 8ccf5a4ac450414c9d3579bd5800b770; no voters:
I20240503 13:56:53.382287 20870 raft_consensus.cc:2781] T 5977344952664d0c8165dcd1f3cea068 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:53.382370 20870 raft_consensus.cc:686] T 5977344952664d0c8165dcd1f3cea068 P 60ba3ffe3b7746469834cd91c84be14e [term 1 LEADER]: Becoming Leader. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Running, Role: LEADER
I20240503 13:56:53.382498 20870 consensus_queue.cc:237] T 5977344952664d0c8165dcd1f3cea068 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:53.382616 15492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5977344952664d0c8165dcd1f3cea068" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546"
I20240503 13:56:53.382725 15492 raft_consensus.cc:3037] T 5977344952664d0c8165dcd1f3cea068 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:53.383652 15492 raft_consensus.cc:2454] T 5977344952664d0c8165dcd1f3cea068 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 1.
I20240503 13:56:53.384510 12912 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:56:53.387526 12904 catalog_manager.cc:5484] T 5977344952664d0c8165dcd1f3cea068 P 60ba3ffe3b7746469834cd91c84be14e reported cstate change: term changed from 0 to 1, leader changed from <none> to 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65). New cstate: current_term: 1 leader_uuid: "60ba3ffe3b7746469834cd91c84be14e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:53.401530 13032 tablet_service.cc:1467] Processing CreateTablet for tablet cf0570d5ce0f42d2bb273f284d589464 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=42d59fce3def4e4196856374f492fb46]), partition=
I20240503 13:56:53.401860 13032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf0570d5ce0f42d2bb273f284d589464. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:53.402446 15867 tablet_service.cc:1467] Processing CreateTablet for tablet cf0570d5ce0f42d2bb273f284d589464 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=42d59fce3def4e4196856374f492fb46]), partition=
I20240503 13:56:53.402657 15867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf0570d5ce0f42d2bb273f284d589464. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:53.404091 24084 tablet_bootstrap.cc:492] T cf0570d5ce0f42d2bb273f284d589464 P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:53.404959 13427 tablet_service.cc:1467] Processing CreateTablet for tablet cf0570d5ce0f42d2bb273f284d589464 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=42d59fce3def4e4196856374f492fb46]), partition=
I20240503 13:56:53.405050 24084 tablet_bootstrap.cc:654] T cf0570d5ce0f42d2bb273f284d589464 P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:53.405207 13427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf0570d5ce0f42d2bb273f284d589464. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:53.405941 24082 tablet_bootstrap.cc:492] T cf0570d5ce0f42d2bb273f284d589464 P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:53.406463 24084 tablet_bootstrap.cc:492] T cf0570d5ce0f42d2bb273f284d589464 P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:53.406553 24084 ts_tablet_manager.cc:1367] T cf0570d5ce0f42d2bb273f284d589464 P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:56:53.407042 24082 tablet_bootstrap.cc:654] T cf0570d5ce0f42d2bb273f284d589464 P e47105a0f46e4511b7112e571688a4c7: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:53.406975 24084 raft_consensus.cc:348] T cf0570d5ce0f42d2bb273f284d589464 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:53.407095 24084 raft_consensus.cc:374] T cf0570d5ce0f42d2bb273f284d589464 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:53.407150 24084 raft_consensus.cc:729] T cf0570d5ce0f42d2bb273f284d589464 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:53.407250 24085 tablet_bootstrap.cc:492] T cf0570d5ce0f42d2bb273f284d589464 P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:53.407267 24084 consensus_queue.cc:260] T cf0570d5ce0f42d2bb273f284d589464 P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:53.407537 24084 ts_tablet_manager.cc:1398] T cf0570d5ce0f42d2bb273f284d589464 P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:53.408135 24085 tablet_bootstrap.cc:654] T cf0570d5ce0f42d2bb273f284d589464 P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:53.414711 24085 tablet_bootstrap.cc:492] T cf0570d5ce0f42d2bb273f284d589464 P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:53.414819 24085 ts_tablet_manager.cc:1367] T cf0570d5ce0f42d2bb273f284d589464 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 13:56:53.415822 24085 raft_consensus.cc:348] T cf0570d5ce0f42d2bb273f284d589464 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:53.415936 24085 raft_consensus.cc:374] T cf0570d5ce0f42d2bb273f284d589464 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:53.415969 24085 raft_consensus.cc:729] T cf0570d5ce0f42d2bb273f284d589464 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:53.416092 24085 consensus_queue.cc:260] T cf0570d5ce0f42d2bb273f284d589464 P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:53.416393 24085 ts_tablet_manager.cc:1398] T cf0570d5ce0f42d2bb273f284d589464 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:53.417604 24082 tablet_bootstrap.cc:492] T cf0570d5ce0f42d2bb273f284d589464 P e47105a0f46e4511b7112e571688a4c7: No bootstrap required, opened a new log
I20240503 13:56:53.417702 24082 ts_tablet_manager.cc:1367] T cf0570d5ce0f42d2bb273f284d589464 P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 13:56:53.418171 24082 raft_consensus.cc:348] T cf0570d5ce0f42d2bb273f284d589464 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:53.418282 24082 raft_consensus.cc:374] T cf0570d5ce0f42d2bb273f284d589464 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:53.418313 24082 raft_consensus.cc:729] T cf0570d5ce0f42d2bb273f284d589464 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:53.418440 24082 consensus_queue.cc:260] T cf0570d5ce0f42d2bb273f284d589464 P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:53.418728 24082 ts_tablet_manager.cc:1398] T cf0570d5ce0f42d2bb273f284d589464 P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:53.463606 13032 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:56:53.466598 15473 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:56:53.470964 15604 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 13:56:53.472769 15867 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:56:53.480662 13427 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:53.488692 13163 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:56:53.497272 15736 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:56:53.514786 13295 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 13:56:53.561700 13315 consensus_queue.cc:237] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:53.562958 15888 raft_consensus.cc:1261] T 866723d26350414197ec27544277db9c P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:56:53.562976 13182 raft_consensus.cc:1261] T 866723d26350414197ec27544277db9c P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:56:53.563338 24059 consensus_queue.cc:1035] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:53.564936 23473 consensus_queue.cc:1035] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:53.566936 23473 raft_consensus.cc:2932] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:53.567387 13182 raft_consensus.cc:2932] T 866723d26350414197ec27544277db9c P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:53.570209 12906 catalog_manager.cc:5484] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 reported cstate change: config changed from index 4 to 5, VOTER 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68) evicted. New cstate: current_term: 1 leader_uuid: "ba0194098aa84449ae17e8a0d4c43703" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:53.574120 13427 tablet_service.cc:1514] Processing DeleteTablet for tablet 866723d26350414197ec27544277db9c with delete_type TABLET_DATA_TOMBSTONED (TS 7690ee79577c45e3a2c8f8b552c0bff8 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:33678
I20240503 13:56:53.576728 24200 tablet_replica.cc:330] T 866723d26350414197ec27544277db9c P 7690ee79577c45e3a2c8f8b552c0bff8: stopping tablet replica
I20240503 13:56:53.576890 24200 raft_consensus.cc:2229] T 866723d26350414197ec27544277db9c P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:53.577059 24200 raft_consensus.cc:2258] T 866723d26350414197ec27544277db9c P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:53.577637 24200 ts_tablet_manager.cc:1875] T 866723d26350414197ec27544277db9c P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:56:53.577837 15888 raft_consensus.cc:2932] T 866723d26350414197ec27544277db9c P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:53.580847 24200 ts_tablet_manager.cc:1888] T 866723d26350414197ec27544277db9c P 7690ee79577c45e3a2c8f8b552c0bff8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 13:56:53.580978 24200 log.cc:1196] T 866723d26350414197ec27544277db9c P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal/wals/866723d26350414197ec27544277db9c
I20240503 13:56:53.581662 12896 catalog_manager.cc:4830] TS 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109): tablet 866723d26350414197ec27544277db9c (table rebalance_test_table_17 [id=52f6b247496e4c3eb63eef4f16160012]) successfully deleted
I20240503 13:56:53.584934 22738 raft_consensus.cc:482] T cf0570d5ce0f42d2bb273f284d589464 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:53.585045 22738 raft_consensus.cc:504] T cf0570d5ce0f42d2bb273f284d589464 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:53.585426 22738 leader_election.cc:288] T cf0570d5ce0f42d2bb273f284d589464 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:53.585887 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf0570d5ce0f42d2bb273f284d589464" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
I20240503 13:56:53.585894 15888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf0570d5ce0f42d2bb273f284d589464" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7" is_pre_election: true
I20240503 13:56:53.586012 15888 raft_consensus.cc:2454] T cf0570d5ce0f42d2bb273f284d589464 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 0.
I20240503 13:56:53.586014 13445 raft_consensus.cc:2454] T cf0570d5ce0f42d2bb273f284d589464 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 0.
I20240503 13:56:53.586274 12987 leader_election.cc:302] T cf0570d5ce0f42d2bb273f284d589464 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 7690ee79577c45e3a2c8f8b552c0bff8; no voters:
I20240503 13:56:53.586526 22738 raft_consensus.cc:2781] T cf0570d5ce0f42d2bb273f284d589464 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:53.586584 22738 raft_consensus.cc:482] T cf0570d5ce0f42d2bb273f284d589464 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:53.586621 22738 raft_consensus.cc:3037] T cf0570d5ce0f42d2bb273f284d589464 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:53.587460 22738 raft_consensus.cc:504] T cf0570d5ce0f42d2bb273f284d589464 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:53.587803 22738 leader_election.cc:288] T cf0570d5ce0f42d2bb273f284d589464 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 election: Requested vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:53.588104 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf0570d5ce0f42d2bb273f284d589464" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
I20240503 13:56:53.588223 13445 raft_consensus.cc:3037] T cf0570d5ce0f42d2bb273f284d589464 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:53.588194 15888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf0570d5ce0f42d2bb273f284d589464" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7"
I20240503 13:56:53.588301 15888 raft_consensus.cc:3037] T cf0570d5ce0f42d2bb273f284d589464 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:53.589258 15888 raft_consensus.cc:2454] T cf0570d5ce0f42d2bb273f284d589464 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 1.
I20240503 13:56:53.589278 13445 raft_consensus.cc:2454] T cf0570d5ce0f42d2bb273f284d589464 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 1.
I20240503 13:56:53.589886 12987 leader_election.cc:302] T cf0570d5ce0f42d2bb273f284d589464 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:53.590065 22738 raft_consensus.cc:2781] T cf0570d5ce0f42d2bb273f284d589464 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:53.590148 22738 raft_consensus.cc:686] T cf0570d5ce0f42d2bb273f284d589464 P 60ba3ffe3b7746469834cd91c84be14e [term 1 LEADER]: Becoming Leader. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Running, Role: LEADER
I20240503 13:56:53.590266 22738 consensus_queue.cc:237] T cf0570d5ce0f42d2bb273f284d589464 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:53.593202 12906 catalog_manager.cc:5484] T cf0570d5ce0f42d2bb273f284d589464 P 60ba3ffe3b7746469834cd91c84be14e reported cstate change: term changed from 0 to 1, leader changed from <none> to 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65). New cstate: current_term: 1 leader_uuid: "60ba3ffe3b7746469834cd91c84be14e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } }
I20240503 13:56:53.627858 13182 consensus_queue.cc:237] T df28831df92d4c12b63b2f7da6c6449a P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:53.629277 13051 raft_consensus.cc:1261] T df28831df92d4c12b63b2f7da6c6449a P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Refusing update from remote peer 8ccf5a4ac450414c9d3579bd5800b770: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 13:56:53.629475 13121 consensus_peers.cc:485] T df28831df92d4c12b63b2f7da6c6449a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885): Couldn't send request to peer 76c386bef61a4bd5a907266a5118ffe2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: df28831df92d4c12b63b2f7da6c6449a. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:53.629403 13315 raft_consensus.cc:1261] T df28831df92d4c12b63b2f7da6c6449a P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Refusing update from remote peer 8ccf5a4ac450414c9d3579bd5800b770: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:56:53.629758 24235 consensus_queue.cc:1035] T df28831df92d4c12b63b2f7da6c6449a P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:53.629935 23843 consensus_queue.cc:1035] T df28831df92d4c12b63b2f7da6c6449a P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:53.632066 23843 raft_consensus.cc:2932] T df28831df92d4c12b63b2f7da6c6449a P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:53.632473 13315 raft_consensus.cc:2932] T df28831df92d4c12b63b2f7da6c6449a P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:53.636150 12904 catalog_manager.cc:5484] T df28831df92d4c12b63b2f7da6c6449a P ba0194098aa84449ae17e8a0d4c43703 reported cstate change: config changed from index -1 to 4, VOTER 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71) added. New cstate: current_term: 1 leader_uuid: "8ccf5a4ac450414c9d3579bd5800b770" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:53.639871 13051 raft_consensus.cc:2932] T df28831df92d4c12b63b2f7da6c6449a P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:53.656932 13182 consensus_queue.cc:237] T 30d2b5ede30744cfb46ec1be232a1405 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:53.658286 13051 raft_consensus.cc:1261] T 30d2b5ede30744cfb46ec1be232a1405 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Refusing update from remote peer 8ccf5a4ac450414c9d3579bd5800b770: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:56:53.658408 13316 raft_consensus.cc:1261] T 30d2b5ede30744cfb46ec1be232a1405 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Refusing update from remote peer 8ccf5a4ac450414c9d3579bd5800b770: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 13:56:53.658479 13118 consensus_peers.cc:485] T 30d2b5ede30744cfb46ec1be232a1405 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951): Couldn't send request to peer 009f849ee7b54c80a3f5264695047e4d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 30d2b5ede30744cfb46ec1be232a1405. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:53.658789 23839 consensus_queue.cc:1035] T 30d2b5ede30744cfb46ec1be232a1405 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:53.658989 24243 consensus_queue.cc:1035] T 30d2b5ede30744cfb46ec1be232a1405 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:53.660887 23839 raft_consensus.cc:2932] T 30d2b5ede30744cfb46ec1be232a1405 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 009f849ee7b54c80a3f5264695047e4d (127.6.98.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:53.661331 13316 raft_consensus.cc:2932] T 30d2b5ede30744cfb46ec1be232a1405 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 009f849ee7b54c80a3f5264695047e4d (127.6.98.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:53.663229 13051 raft_consensus.cc:2932] T 30d2b5ede30744cfb46ec1be232a1405 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 009f849ee7b54c80a3f5264695047e4d (127.6.98.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:53.664413 12904 catalog_manager.cc:5484] T 30d2b5ede30744cfb46ec1be232a1405 P 8ccf5a4ac450414c9d3579bd5800b770 reported cstate change: config changed from index -1 to 4, VOTER 009f849ee7b54c80a3f5264695047e4d (127.6.98.70) added. New cstate: current_term: 1 leader_uuid: "8ccf5a4ac450414c9d3579bd5800b770" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:53.716223 12907 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:56:53.725260 13445 raft_consensus.cc:1261] T cf0570d5ce0f42d2bb273f284d589464 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer 60ba3ffe3b7746469834cd91c84be14e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:53.725417 15889 raft_consensus.cc:1261] T cf0570d5ce0f42d2bb273f284d589464 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Refusing update from remote peer 60ba3ffe3b7746469834cd91c84be14e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:53.725927 20870 consensus_queue.cc:1035] T cf0570d5ce0f42d2bb273f284d589464 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:53.726258 20870 consensus_queue.cc:1035] T cf0570d5ce0f42d2bb273f284d589464 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:53.731185 12907 catalog_manager.cc:5576] TS e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753) does not have the latest schema for tablet cf0570d5ce0f42d2bb273f284d589464 (table rebalancer_dynamic_table [id=42d59fce3def4e4196856374f492fb46]). Expected version 1 got 0
I20240503 13:56:53.731611 12904 catalog_manager.cc:5576] TS 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109) does not have the latest schema for tablet cf0570d5ce0f42d2bb273f284d589464 (table rebalancer_dynamic_table [id=42d59fce3def4e4196856374f492fb46]). Expected version 1 got 0
I20240503 13:56:53.734629 24304 tablet.cc:1722] T cf0570d5ce0f42d2bb273f284d589464 P 60ba3ffe3b7746469834cd91c84be14e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:53.735054 24304 tablet.cc:1620] T cf0570d5ce0f42d2bb273f284d589464 P 60ba3ffe3b7746469834cd91c84be14e: MemRowSet was empty: no flush needed.
I20240503 13:56:53.737330 12907 catalog_manager.cc:5576] TS e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753) does not have the latest schema for tablet cf0570d5ce0f42d2bb273f284d589464 (table rebalancer_dynamic_table [id=42d59fce3def4e4196856374f492fb46]). Expected version 1 got 0
I20240503 13:56:53.742344 24306 tablet.cc:1722] T cf0570d5ce0f42d2bb273f284d589464 P 7690ee79577c45e3a2c8f8b552c0bff8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:53.742653 24306 tablet.cc:1620] T cf0570d5ce0f42d2bb273f284d589464 P 7690ee79577c45e3a2c8f8b552c0bff8: MemRowSet was empty: no flush needed.
I20240503 13:56:53.743583 12905 catalog_manager.cc:5576] TS 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109) does not have the latest schema for tablet cf0570d5ce0f42d2bb273f284d589464 (table rebalancer_dynamic_table [id=42d59fce3def4e4196856374f492fb46]). Expected version 1 got 0
I20240503 13:56:53.746212 12904 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=42d59fce3def4e4196856374f492fb46] alter complete (version 1)
I20240503 13:56:53.754897 24305 tablet.cc:1722] T cf0570d5ce0f42d2bb273f284d589464 P e47105a0f46e4511b7112e571688a4c7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:53.755595 24305 tablet.cc:1620] T cf0570d5ce0f42d2bb273f284d589464 P e47105a0f46e4511b7112e571688a4c7: MemRowSet was empty: no flush needed.
I20240503 13:56:53.817806 15604 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 13:56:53.821429 15473 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:56:53.825140 13032 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 13:56:53.831087 15736 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:56:53.834578 15867 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:56:53.838321 13427 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:53.838546 13295 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 13:56:53.856720 23094 consensus_queue.cc:1035] T fc7c89676b1b4262a7579483ccfe8991 P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:53.856971 13163 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:56:53.857172 12904 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:56:53.864799 24304 tablet.cc:1722] T cf0570d5ce0f42d2bb273f284d589464 P 60ba3ffe3b7746469834cd91c84be14e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:53.865168 24304 tablet.cc:1620] T cf0570d5ce0f42d2bb273f284d589464 P 60ba3ffe3b7746469834cd91c84be14e: MemRowSet was empty: no flush needed.
I20240503 13:56:53.865835 24306 tablet.cc:1722] T cf0570d5ce0f42d2bb273f284d589464 P 7690ee79577c45e3a2c8f8b552c0bff8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:53.866117 24306 tablet.cc:1620] T cf0570d5ce0f42d2bb273f284d589464 P 7690ee79577c45e3a2c8f8b552c0bff8: MemRowSet was empty: no flush needed.
I20240503 13:56:53.867825 24305 tablet.cc:1722] T cf0570d5ce0f42d2bb273f284d589464 P e47105a0f46e4511b7112e571688a4c7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:53.868120 24305 tablet.cc:1620] T cf0570d5ce0f42d2bb273f284d589464 P e47105a0f46e4511b7112e571688a4c7: MemRowSet was empty: no flush needed.
I20240503 13:56:53.872205 12904 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=42d59fce3def4e4196856374f492fb46] alter complete (version 2)
I20240503 13:56:53.880199 20870 consensus_queue.cc:1035] T 5977344952664d0c8165dcd1f3cea068 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:53.882678 23686 consensus_queue.cc:1035] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:53.885934 23686 consensus_queue.cc:1035] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:53.902944 20870 consensus_queue.cc:1035] T 5977344952664d0c8165dcd1f3cea068 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:53.905050 20722 consensus_queue.cc:1035] T fc7c89676b1b4262a7579483ccfe8991 P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:53.932458 12905 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_extra_table_64"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:56:53.939265 15736 tablet_service.cc:1467] Processing CreateTablet for tablet 93d1e77b0d1d410093cb9233f03fa70e (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=ec2f1f5bade947628f299aa40755dac5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:53.939612 15736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93d1e77b0d1d410093cb9233f03fa70e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:53.941932 15604 tablet_service.cc:1467] Processing CreateTablet for tablet dd39f3ab7cdb42b38267214056605328 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=ec2f1f5bade947628f299aa40755dac5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:53.942133 15867 tablet_service.cc:1467] Processing CreateTablet for tablet dd39f3ab7cdb42b38267214056605328 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=ec2f1f5bade947628f299aa40755dac5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:53.942276 15604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd39f3ab7cdb42b38267214056605328. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:53.942392 15867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd39f3ab7cdb42b38267214056605328. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:53.942392 13295 tablet_service.cc:1467] Processing CreateTablet for tablet 93d1e77b0d1d410093cb9233f03fa70e (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=ec2f1f5bade947628f299aa40755dac5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:53.942641 13295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93d1e77b0d1d410093cb9233f03fa70e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:53.944777 13296 tablet_service.cc:1467] Processing CreateTablet for tablet e4e7d7a8644946a59bf5a3df7def80f4 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=ec2f1f5bade947628f299aa40755dac5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:53.945008 13296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4e7d7a8644946a59bf5a3df7def80f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:53.945519 24349 tablet_bootstrap.cc:492] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:53.946406 24349 tablet_bootstrap.cc:654] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:53.947280 13032 tablet_service.cc:1467] Processing CreateTablet for tablet e4e7d7a8644946a59bf5a3df7def80f4 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=ec2f1f5bade947628f299aa40755dac5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:53.947396 24349 tablet_bootstrap.cc:492] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:53.947470 24349 ts_tablet_manager.cc:1367] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:56:53.947567 13032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4e7d7a8644946a59bf5a3df7def80f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:53.947921 24349 raft_consensus.cc:348] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:53.948019 24349 raft_consensus.cc:374] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:53.948048 24349 raft_consensus.cc:729] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:53.948155 24349 consensus_queue.cc:260] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:53.948421 24349 ts_tablet_manager.cc:1398] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:53.950065 13034 tablet_service.cc:1467] Processing CreateTablet for tablet dd39f3ab7cdb42b38267214056605328 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=ec2f1f5bade947628f299aa40755dac5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:53.950287 13034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd39f3ab7cdb42b38267214056605328. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:53.950840 24353 tablet_bootstrap.cc:492] T e4e7d7a8644946a59bf5a3df7def80f4 P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:53.951803 24353 tablet_bootstrap.cc:654] T e4e7d7a8644946a59bf5a3df7def80f4 P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:53.952468 15738 tablet_service.cc:1467] Processing CreateTablet for tablet e4e7d7a8644946a59bf5a3df7def80f4 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=ec2f1f5bade947628f299aa40755dac5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:53.952762 15738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4e7d7a8644946a59bf5a3df7def80f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:53.953728 24353 tablet_bootstrap.cc:492] T e4e7d7a8644946a59bf5a3df7def80f4 P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:53.953792 24353 ts_tablet_manager.cc:1367] T e4e7d7a8644946a59bf5a3df7def80f4 P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:56:53.954401 24353 raft_consensus.cc:348] T e4e7d7a8644946a59bf5a3df7def80f4 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:53.954499 24353 raft_consensus.cc:374] T e4e7d7a8644946a59bf5a3df7def80f4 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:53.954528 24353 raft_consensus.cc:729] T e4e7d7a8644946a59bf5a3df7def80f4 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:53.954643 24353 consensus_queue.cc:260] T e4e7d7a8644946a59bf5a3df7def80f4 P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:53.955097 13163 tablet_service.cc:1467] Processing CreateTablet for tablet 93d1e77b0d1d410093cb9233f03fa70e (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=ec2f1f5bade947628f299aa40755dac5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:53.955346 13163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93d1e77b0d1d410093cb9233f03fa70e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:53.955642 24353 ts_tablet_manager.cc:1398] T e4e7d7a8644946a59bf5a3df7def80f4 P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:53.956954 24348 tablet_bootstrap.cc:492] T 93d1e77b0d1d410093cb9233f03fa70e P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:53.957614 24359 tablet_bootstrap.cc:492] T 93d1e77b0d1d410093cb9233f03fa70e P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:53.958165 24348 tablet_bootstrap.cc:654] T 93d1e77b0d1d410093cb9233f03fa70e P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:53.956665 24350 tablet_bootstrap.cc:492] T dd39f3ab7cdb42b38267214056605328 P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:53.958617 24359 tablet_bootstrap.cc:654] T 93d1e77b0d1d410093cb9233f03fa70e P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:53.959419 24350 tablet_bootstrap.cc:654] T dd39f3ab7cdb42b38267214056605328 P e47105a0f46e4511b7112e571688a4c7: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:53.959872 24353 tablet_bootstrap.cc:492] T dd39f3ab7cdb42b38267214056605328 P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:53.960850 24353 tablet_bootstrap.cc:654] T dd39f3ab7cdb42b38267214056605328 P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:53.961405 24348 tablet_bootstrap.cc:492] T 93d1e77b0d1d410093cb9233f03fa70e P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:53.961493 24348 ts_tablet_manager.cc:1367] T 93d1e77b0d1d410093cb9233f03fa70e P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:56:53.961889 24350 tablet_bootstrap.cc:492] T dd39f3ab7cdb42b38267214056605328 P e47105a0f46e4511b7112e571688a4c7: No bootstrap required, opened a new log
I20240503 13:56:53.961961 24353 tablet_bootstrap.cc:492] T dd39f3ab7cdb42b38267214056605328 P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:53.961978 24350 ts_tablet_manager.cc:1367] T dd39f3ab7cdb42b38267214056605328 P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240503 13:56:53.962244 24353 ts_tablet_manager.cc:1367] T dd39f3ab7cdb42b38267214056605328 P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:56:53.962391 24350 raft_consensus.cc:348] T dd39f3ab7cdb42b38267214056605328 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:53.962965 24348 raft_consensus.cc:348] T 93d1e77b0d1d410093cb9233f03fa70e P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:53.963033 24353 raft_consensus.cc:348] T dd39f3ab7cdb42b38267214056605328 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:53.963136 24350 raft_consensus.cc:374] T dd39f3ab7cdb42b38267214056605328 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:53.963199 24350 raft_consensus.cc:729] T dd39f3ab7cdb42b38267214056605328 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:53.963223 24353 raft_consensus.cc:374] T dd39f3ab7cdb42b38267214056605328 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:53.963301 24353 raft_consensus.cc:729] T dd39f3ab7cdb42b38267214056605328 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:53.963443 24353 consensus_queue.cc:260] T dd39f3ab7cdb42b38267214056605328 P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:53.963284 24348 raft_consensus.cc:374] T 93d1e77b0d1d410093cb9233f03fa70e P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:53.964151 24348 raft_consensus.cc:729] T 93d1e77b0d1d410093cb9233f03fa70e P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:53.964470 24348 consensus_queue.cc:260] T 93d1e77b0d1d410093cb9233f03fa70e P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:53.964054 24350 consensus_queue.cc:260] T dd39f3ab7cdb42b38267214056605328 P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:53.964726 24352 tablet_bootstrap.cc:492] T e4e7d7a8644946a59bf5a3df7def80f4 P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:53.965103 24348 ts_tablet_manager.cc:1398] T 93d1e77b0d1d410093cb9233f03fa70e P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 13:56:53.965391 24350 ts_tablet_manager.cc:1398] T dd39f3ab7cdb42b38267214056605328 P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 13:56:53.965662 24348 tablet_bootstrap.cc:492] T e4e7d7a8644946a59bf5a3df7def80f4 P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:53.966811 24348 tablet_bootstrap.cc:654] T e4e7d7a8644946a59bf5a3df7def80f4 P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:53.967166 24352 tablet_bootstrap.cc:654] T e4e7d7a8644946a59bf5a3df7def80f4 P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:53.968683 24348 tablet_bootstrap.cc:492] T e4e7d7a8644946a59bf5a3df7def80f4 P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:53.968765 24348 ts_tablet_manager.cc:1367] T e4e7d7a8644946a59bf5a3df7def80f4 P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 13:56:53.969206 24348 raft_consensus.cc:348] T e4e7d7a8644946a59bf5a3df7def80f4 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:53.969308 24348 raft_consensus.cc:374] T e4e7d7a8644946a59bf5a3df7def80f4 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:53.969336 24348 raft_consensus.cc:729] T e4e7d7a8644946a59bf5a3df7def80f4 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:53.969455 24348 consensus_queue.cc:260] T e4e7d7a8644946a59bf5a3df7def80f4 P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:53.969615 24352 tablet_bootstrap.cc:492] T e4e7d7a8644946a59bf5a3df7def80f4 P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:53.969694 24352 ts_tablet_manager.cc:1367] T e4e7d7a8644946a59bf5a3df7def80f4 P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:56:53.969736 24348 ts_tablet_manager.cc:1398] T e4e7d7a8644946a59bf5a3df7def80f4 P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:53.970228 24352 raft_consensus.cc:348] T e4e7d7a8644946a59bf5a3df7def80f4 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:53.970481 24352 raft_consensus.cc:374] T e4e7d7a8644946a59bf5a3df7def80f4 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:53.970577 24352 raft_consensus.cc:729] T e4e7d7a8644946a59bf5a3df7def80f4 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:53.970752 24352 consensus_queue.cc:260] T e4e7d7a8644946a59bf5a3df7def80f4 P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:53.971088 24352 ts_tablet_manager.cc:1398] T e4e7d7a8644946a59bf5a3df7def80f4 P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:53.972035 24352 tablet_bootstrap.cc:492] T 93d1e77b0d1d410093cb9233f03fa70e P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:53.973008 24352 tablet_bootstrap.cc:654] T 93d1e77b0d1d410093cb9233f03fa70e P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:53.975399 24353 ts_tablet_manager.cc:1398] T dd39f3ab7cdb42b38267214056605328 P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.013s user 0.001s sys 0.000s
I20240503 13:56:53.976882 24359 tablet_bootstrap.cc:492] T 93d1e77b0d1d410093cb9233f03fa70e P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:53.977002 24359 ts_tablet_manager.cc:1367] T 93d1e77b0d1d410093cb9233f03fa70e P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240503 13:56:53.977507 24359 raft_consensus.cc:348] T 93d1e77b0d1d410093cb9233f03fa70e P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:53.977627 24359 raft_consensus.cc:374] T 93d1e77b0d1d410093cb9233f03fa70e P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:53.977655 24359 raft_consensus.cc:729] T 93d1e77b0d1d410093cb9233f03fa70e P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:53.977773 24359 consensus_queue.cc:260] T 93d1e77b0d1d410093cb9233f03fa70e P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:53.978073 24359 ts_tablet_manager.cc:1398] T 93d1e77b0d1d410093cb9233f03fa70e P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:53.979068 12906 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:56:53.981547 23686 raft_consensus.cc:482] T e4e7d7a8644946a59bf5a3df7def80f4 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:53.981647 23686 raft_consensus.cc:504] T e4e7d7a8644946a59bf5a3df7def80f4 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:53.981972 23686 leader_election.cc:288] T e4e7d7a8644946a59bf5a3df7def80f4 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279), 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:53.983880 13316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e7d7a8644946a59bf5a3df7def80f4" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703" is_pre_election: true
I20240503 13:56:53.984012 13316 raft_consensus.cc:2454] T e4e7d7a8644946a59bf5a3df7def80f4 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 0.
I20240503 13:56:53.985095 15694 leader_election.cc:302] T e4e7d7a8644946a59bf5a3df7def80f4 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76c386bef61a4bd5a907266a5118ffe2, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:53.985497 23131 raft_consensus.cc:2781] T e4e7d7a8644946a59bf5a3df7def80f4 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:53.985564 23131 raft_consensus.cc:482] T e4e7d7a8644946a59bf5a3df7def80f4 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:53.985601 23131 raft_consensus.cc:3037] T e4e7d7a8644946a59bf5a3df7def80f4 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:53.985594 24304 tablet.cc:1722] T cf0570d5ce0f42d2bb273f284d589464 P 60ba3ffe3b7746469834cd91c84be14e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:53.986742 23131 raft_consensus.cc:504] T e4e7d7a8644946a59bf5a3df7def80f4 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:53.987139 23131 leader_election.cc:288] T e4e7d7a8644946a59bf5a3df7def80f4 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 election: Requested vote from peers ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279), 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:53.988072 24305 tablet.cc:1722] T cf0570d5ce0f42d2bb273f284d589464 P e47105a0f46e4511b7112e571688a4c7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:53.988353 13051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e7d7a8644946a59bf5a3df7def80f4" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e" is_pre_election: true
I20240503 13:56:53.988363 13315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e7d7a8644946a59bf5a3df7def80f4" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703"
I20240503 13:56:53.988500 13051 raft_consensus.cc:2454] T e4e7d7a8644946a59bf5a3df7def80f4 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 0.
I20240503 13:56:53.988514 13315 raft_consensus.cc:3037] T e4e7d7a8644946a59bf5a3df7def80f4 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:53.989286 13051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e7d7a8644946a59bf5a3df7def80f4" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e"
I20240503 13:56:53.989387 13051 raft_consensus.cc:3037] T e4e7d7a8644946a59bf5a3df7def80f4 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:53.989485 13315 raft_consensus.cc:2454] T e4e7d7a8644946a59bf5a3df7def80f4 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 1.
I20240503 13:56:53.990305 13051 raft_consensus.cc:2454] T e4e7d7a8644946a59bf5a3df7def80f4 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 1.
I20240503 13:56:53.990938 15694 leader_election.cc:302] T e4e7d7a8644946a59bf5a3df7def80f4 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76c386bef61a4bd5a907266a5118ffe2, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:53.991232 24352 tablet_bootstrap.cc:492] T 93d1e77b0d1d410093cb9233f03fa70e P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:53.991333 24352 ts_tablet_manager.cc:1367] T 93d1e77b0d1d410093cb9233f03fa70e P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240503 13:56:53.992131 23131 raft_consensus.cc:2781] T e4e7d7a8644946a59bf5a3df7def80f4 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:53.992339 23131 raft_consensus.cc:686] T e4e7d7a8644946a59bf5a3df7def80f4 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 LEADER]: Becoming Leader. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Running, Role: LEADER
I20240503 13:56:53.992311 24352 raft_consensus.cc:348] T 93d1e77b0d1d410093cb9233f03fa70e P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:53.992409 24352 raft_consensus.cc:374] T 93d1e77b0d1d410093cb9233f03fa70e P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:53.992439 24352 raft_consensus.cc:729] T 93d1e77b0d1d410093cb9233f03fa70e P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:53.992574 24352 consensus_queue.cc:260] T 93d1e77b0d1d410093cb9233f03fa70e P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:53.992659 23131 consensus_queue.cc:237] T e4e7d7a8644946a59bf5a3df7def80f4 P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:53.992897 24352 ts_tablet_manager.cc:1398] T 93d1e77b0d1d410093cb9233f03fa70e P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:53.996053 12904 catalog_manager.cc:5484] T e4e7d7a8644946a59bf5a3df7def80f4 P 76c386bef61a4bd5a907266a5118ffe2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71). New cstate: current_term: 1 leader_uuid: "76c386bef61a4bd5a907266a5118ffe2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } }
I20240503 13:56:53.997016 24306 tablet.cc:1722] T cf0570d5ce0f42d2bb273f284d589464 P 7690ee79577c45e3a2c8f8b552c0bff8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:53.997897 12906 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table_new_name [id=42d59fce3def4e4196856374f492fb46] alter complete (version 3)
I20240503 13:56:54.088249 22256 raft_consensus.cc:482] T dd39f3ab7cdb42b38267214056605328 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:54.088382 22256 raft_consensus.cc:504] T dd39f3ab7cdb42b38267214056605328 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:54.088824 22256 leader_election.cc:288] T dd39f3ab7cdb42b38267214056605328 P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695), 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951)
I20240503 13:56:54.089123 13051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd39f3ab7cdb42b38267214056605328" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e" is_pre_election: true
I20240503 13:56:54.089270 13051 raft_consensus.cc:2454] T dd39f3ab7cdb42b38267214056605328 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 0.
I20240503 13:56:54.089347 15623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd39f3ab7cdb42b38267214056605328" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d" is_pre_election: true
I20240503 13:56:54.089484 15623 raft_consensus.cc:2454] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 0.
I20240503 13:56:54.089509 15823 leader_election.cc:302] T dd39f3ab7cdb42b38267214056605328 P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:54.089741 22256 raft_consensus.cc:2781] T dd39f3ab7cdb42b38267214056605328 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:54.089807 22256 raft_consensus.cc:482] T dd39f3ab7cdb42b38267214056605328 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:54.089854 22256 raft_consensus.cc:3037] T dd39f3ab7cdb42b38267214056605328 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:54.090885 22256 raft_consensus.cc:504] T dd39f3ab7cdb42b38267214056605328 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:54.091217 22256 leader_election.cc:288] T dd39f3ab7cdb42b38267214056605328 P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 election: Requested vote from peers 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695), 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951)
I20240503 13:56:54.091495 15623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd39f3ab7cdb42b38267214056605328" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d"
I20240503 13:56:54.091572 13051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd39f3ab7cdb42b38267214056605328" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e"
I20240503 13:56:54.091606 15623 raft_consensus.cc:3037] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:54.091677 13051 raft_consensus.cc:3037] T dd39f3ab7cdb42b38267214056605328 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:54.092592 15623 raft_consensus.cc:2454] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 1.
I20240503 13:56:54.092614 13051 raft_consensus.cc:2454] T dd39f3ab7cdb42b38267214056605328 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 1.
I20240503 13:56:54.092839 15823 leader_election.cc:302] T dd39f3ab7cdb42b38267214056605328 P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:54.092998 22256 raft_consensus.cc:2781] T dd39f3ab7cdb42b38267214056605328 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:54.093113 22256 raft_consensus.cc:686] T dd39f3ab7cdb42b38267214056605328 P e47105a0f46e4511b7112e571688a4c7 [term 1 LEADER]: Becoming Leader. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Running, Role: LEADER
I20240503 13:56:54.093271 22256 consensus_queue.cc:237] T dd39f3ab7cdb42b38267214056605328 P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:54.096696 12909 catalog_manager.cc:5484] T dd39f3ab7cdb42b38267214056605328 P e47105a0f46e4511b7112e571688a4c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e47105a0f46e4511b7112e571688a4c7 (127.6.98.72). New cstate: current_term: 1 leader_uuid: "e47105a0f46e4511b7112e571688a4c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:54.102034 12909 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:54.102160 12909 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:54.104013 12909 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Sending DeleteTablet for 3 replicas of tablet cf0570d5ce0f42d2bb273f284d589464
I20240503 13:56:54.104781 13427 tablet_service.cc:1514] Processing DeleteTablet for tablet cf0570d5ce0f42d2bb273f284d589464 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:54 UTC) from {username='slave'} at 127.0.0.1:33678
I20240503 13:56:54.105096 24397 tablet_replica.cc:330] T cf0570d5ce0f42d2bb273f284d589464 P 7690ee79577c45e3a2c8f8b552c0bff8: stopping tablet replica
I20240503 13:56:54.105273 24397 raft_consensus.cc:2229] T cf0570d5ce0f42d2bb273f284d589464 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:54.105479 13034 tablet_service.cc:1514] Processing DeleteTablet for tablet cf0570d5ce0f42d2bb273f284d589464 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:54 UTC) from {username='slave'} at 127.0.0.1:59458
I20240503 13:56:54.105814 24397 raft_consensus.cc:2258] T cf0570d5ce0f42d2bb273f284d589464 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:54.105835 24398 tablet_replica.cc:330] T cf0570d5ce0f42d2bb273f284d589464 P 60ba3ffe3b7746469834cd91c84be14e: stopping tablet replica
I20240503 13:56:54.106190 24398 raft_consensus.cc:2229] T cf0570d5ce0f42d2bb273f284d589464 P 60ba3ffe3b7746469834cd91c84be14e [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:56:54.106387 24397 ts_tablet_manager.cc:1875] T cf0570d5ce0f42d2bb273f284d589464 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:54.106429 24398 raft_consensus.cc:2258] T cf0570d5ce0f42d2bb273f284d589464 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:54.106962 24398 ts_tablet_manager.cc:1875] T cf0570d5ce0f42d2bb273f284d589464 P 60ba3ffe3b7746469834cd91c84be14e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:54.108665 15867 tablet_service.cc:1514] Processing DeleteTablet for tablet cf0570d5ce0f42d2bb273f284d589464 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:54 UTC) from {username='slave'} at 127.0.0.1:50498
I20240503 13:56:54.109066 24397 ts_tablet_manager.cc:1888] T cf0570d5ce0f42d2bb273f284d589464 P 7690ee79577c45e3a2c8f8b552c0bff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:54.109158 24397 log.cc:1196] T cf0570d5ce0f42d2bb273f284d589464 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal/wals/cf0570d5ce0f42d2bb273f284d589464
I20240503 13:56:54.109330 24398 ts_tablet_manager.cc:1888] T cf0570d5ce0f42d2bb273f284d589464 P 60ba3ffe3b7746469834cd91c84be14e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:54.109412 24398 log.cc:1196] T cf0570d5ce0f42d2bb273f284d589464 P 60ba3ffe3b7746469834cd91c84be14e: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal/wals/cf0570d5ce0f42d2bb273f284d589464
I20240503 13:56:54.109443 24397 ts_tablet_manager.cc:1909] T cf0570d5ce0f42d2bb273f284d589464 P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting consensus metadata
I20240503 13:56:54.109680 24398 ts_tablet_manager.cc:1909] T cf0570d5ce0f42d2bb273f284d589464 P 60ba3ffe3b7746469834cd91c84be14e: Deleting consensus metadata
I20240503 13:56:54.110248 12896 catalog_manager.cc:4830] TS 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109): tablet cf0570d5ce0f42d2bb273f284d589464 (table rebalancer_dynamic_table_new_name [id=42d59fce3def4e4196856374f492fb46]) successfully deleted
I20240503 13:56:54.110505 12897 catalog_manager.cc:4830] TS 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695): tablet cf0570d5ce0f42d2bb273f284d589464 (table rebalancer_dynamic_table_new_name [id=42d59fce3def4e4196856374f492fb46]) successfully deleted
I20240503 13:56:54.119185 24235 raft_consensus.cc:482] T 93d1e77b0d1d410093cb9233f03fa70e P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:54.119302 24235 raft_consensus.cc:504] T 93d1e77b0d1d410093cb9233f03fa70e P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:54.119647 24235 leader_election.cc:288] T 93d1e77b0d1d410093cb9233f03fa70e P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885), ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:54.120112 13315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d1e77b0d1d410093cb9233f03fa70e" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703" is_pre_election: true
I20240503 13:56:54.120268 13315 raft_consensus.cc:2454] T 93d1e77b0d1d410093cb9233f03fa70e P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 0.
I20240503 13:56:54.120293 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d1e77b0d1d410093cb9233f03fa70e" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2" is_pre_election: true
I20240503 13:56:54.120411 15757 raft_consensus.cc:2454] T 93d1e77b0d1d410093cb9233f03fa70e P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 0.
I20240503 13:56:54.120641 24399 tablet_replica.cc:330] T cf0570d5ce0f42d2bb273f284d589464 P e47105a0f46e4511b7112e571688a4c7: stopping tablet replica
I20240503 13:56:54.120770 13121 leader_election.cc:302] T 93d1e77b0d1d410093cb9233f03fa70e P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76c386bef61a4bd5a907266a5118ffe2, 8ccf5a4ac450414c9d3579bd5800b770; no voters:
I20240503 13:56:54.120813 24399 raft_consensus.cc:2229] T cf0570d5ce0f42d2bb273f284d589464 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:54.120939 24399 raft_consensus.cc:2258] T cf0570d5ce0f42d2bb273f284d589464 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:54.121052 23843 raft_consensus.cc:2781] T 93d1e77b0d1d410093cb9233f03fa70e P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:54.121107 23843 raft_consensus.cc:482] T 93d1e77b0d1d410093cb9233f03fa70e P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:54.121156 23843 raft_consensus.cc:3037] T 93d1e77b0d1d410093cb9233f03fa70e P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:54.121598 24399 ts_tablet_manager.cc:1875] T cf0570d5ce0f42d2bb273f284d589464 P e47105a0f46e4511b7112e571688a4c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:54.122126 23843 raft_consensus.cc:504] T 93d1e77b0d1d410093cb9233f03fa70e P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:54.122473 23843 leader_election.cc:288] T 93d1e77b0d1d410093cb9233f03fa70e P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 election: Requested vote from peers 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885), ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:54.123343 13315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d1e77b0d1d410093cb9233f03fa70e" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703"
I20240503 13:56:54.123471 13315 raft_consensus.cc:3037] T 93d1e77b0d1d410093cb9233f03fa70e P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:54.124217 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d1e77b0d1d410093cb9233f03fa70e" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2"
I20240503 13:56:54.124330 15757 raft_consensus.cc:3037] T 93d1e77b0d1d410093cb9233f03fa70e P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:54.124526 13315 raft_consensus.cc:2454] T 93d1e77b0d1d410093cb9233f03fa70e P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 1.
I20240503 13:56:54.124536 24399 ts_tablet_manager.cc:1888] T cf0570d5ce0f42d2bb273f284d589464 P e47105a0f46e4511b7112e571688a4c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:54.124624 24399 log.cc:1196] T cf0570d5ce0f42d2bb273f284d589464 P e47105a0f46e4511b7112e571688a4c7: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-7/wal/wals/cf0570d5ce0f42d2bb273f284d589464
I20240503 13:56:54.124864 24399 ts_tablet_manager.cc:1909] T cf0570d5ce0f42d2bb273f284d589464 P e47105a0f46e4511b7112e571688a4c7: Deleting consensus metadata
I20240503 13:56:54.125416 15757 raft_consensus.cc:2454] T 93d1e77b0d1d410093cb9233f03fa70e P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 1.
I20240503 13:56:54.125913 12896 catalog_manager.cc:4830] TS e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): tablet cf0570d5ce0f42d2bb273f284d589464 (table rebalancer_dynamic_table_new_name [id=42d59fce3def4e4196856374f492fb46]) successfully deleted
I20240503 13:56:54.127317 13121 leader_election.cc:302] T 93d1e77b0d1d410093cb9233f03fa70e P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76c386bef61a4bd5a907266a5118ffe2, 8ccf5a4ac450414c9d3579bd5800b770; no voters:
I20240503 13:56:54.127480 23843 raft_consensus.cc:2781] T 93d1e77b0d1d410093cb9233f03fa70e P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:54.127562 23843 raft_consensus.cc:686] T 93d1e77b0d1d410093cb9233f03fa70e P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 LEADER]: Becoming Leader. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Running, Role: LEADER
I20240503 13:56:54.127681 23843 consensus_queue.cc:237] T 93d1e77b0d1d410093cb9233f03fa70e P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:54.131690 12909 catalog_manager.cc:5484] T 93d1e77b0d1d410093cb9233f03fa70e P 8ccf5a4ac450414c9d3579bd5800b770 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66). New cstate: current_term: 1 leader_uuid: "8ccf5a4ac450414c9d3579bd5800b770" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } }
I20240503 13:56:54.151458 13427 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:54.152810 13034 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:56:54.155406 15473 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:56:54.156656 15604 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 13:56:54.169095 24412 ts_tablet_manager.cc:902] T df28831df92d4c12b63b2f7da6c6449a P 76c386bef61a4bd5a907266a5118ffe2: Initiating tablet copy from peer 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:54.169659 24412 tablet_copy_client.cc:308] T df28831df92d4c12b63b2f7da6c6449a P 76c386bef61a4bd5a907266a5118ffe2: tablet copy: Beginning tablet copy session from remote peer at address 127.6.98.66:36387
I20240503 13:56:54.171242 13163 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 13:56:54.171932 13205 tablet_copy_service.cc:140] P 8ccf5a4ac450414c9d3579bd5800b770: Received BeginTabletCopySession request for tablet df28831df92d4c12b63b2f7da6c6449a from peer 76c386bef61a4bd5a907266a5118ffe2 ({username='slave'} at 127.6.98.71:41361)
I20240503 13:56:54.172025 13205 tablet_copy_service.cc:161] P 8ccf5a4ac450414c9d3579bd5800b770: Beginning new tablet copy session on tablet df28831df92d4c12b63b2f7da6c6449a from peer 76c386bef61a4bd5a907266a5118ffe2 at {username='slave'} at 127.6.98.71:41361: session id = 76c386bef61a4bd5a907266a5118ffe2-df28831df92d4c12b63b2f7da6c6449a
I20240503 13:56:54.173087 13205 tablet_copy_source_session.cc:205] T df28831df92d4c12b63b2f7da6c6449a P 8ccf5a4ac450414c9d3579bd5800b770: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:56:54.173656 13295 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 13:56:54.174113 15738 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 13:56:54.174261 24412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df28831df92d4c12b63b2f7da6c6449a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:54.178342 15867 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 13:56:54.209856 12904 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:56:54.212741 24412 tablet_copy_client.cc:783] T df28831df92d4c12b63b2f7da6c6449a P 76c386bef61a4bd5a907266a5118ffe2: tablet copy: Starting download of 0 data blocks...
I20240503 13:56:54.212934 24412 tablet_copy_client.cc:647] T df28831df92d4c12b63b2f7da6c6449a P 76c386bef61a4bd5a907266a5118ffe2: tablet copy: Starting download of 1 WAL segments...
I20240503 13:56:54.214241 24412 tablet_copy_client.cc:523] T df28831df92d4c12b63b2f7da6c6449a P 76c386bef61a4bd5a907266a5118ffe2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:56:54.217103 15473 tablet_service.cc:1467] Processing CreateTablet for tablet 3f44a6a229f7456bafc8b24a837fb68a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=639667bd73754b769569bbc53d4e1d94]), partition=
I20240503 13:56:54.217360 15473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f44a6a229f7456bafc8b24a837fb68a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:54.217905 24412 tablet_bootstrap.cc:492] T df28831df92d4c12b63b2f7da6c6449a P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:54.220934 15604 tablet_service.cc:1467] Processing CreateTablet for tablet 3f44a6a229f7456bafc8b24a837fb68a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=639667bd73754b769569bbc53d4e1d94]), partition=
I20240503 13:56:54.221174 15604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f44a6a229f7456bafc8b24a837fb68a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:54.221592 13295 tablet_service.cc:1467] Processing CreateTablet for tablet 3f44a6a229f7456bafc8b24a837fb68a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=639667bd73754b769569bbc53d4e1d94]), partition=
I20240503 13:56:54.221822 13295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f44a6a229f7456bafc8b24a837fb68a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:54.223217 24349 tablet_bootstrap.cc:492] T 3f44a6a229f7456bafc8b24a837fb68a P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:54.223776 24352 tablet_bootstrap.cc:492] T 3f44a6a229f7456bafc8b24a837fb68a P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:54.224110 24349 tablet_bootstrap.cc:654] T 3f44a6a229f7456bafc8b24a837fb68a P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:54.224704 24352 tablet_bootstrap.cc:654] T 3f44a6a229f7456bafc8b24a837fb68a P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:54.227224 24418 tablet_bootstrap.cc:492] T 3f44a6a229f7456bafc8b24a837fb68a P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:54.227422 24352 tablet_bootstrap.cc:492] T 3f44a6a229f7456bafc8b24a837fb68a P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:54.227509 24352 ts_tablet_manager.cc:1367] T 3f44a6a229f7456bafc8b24a837fb68a P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:56:54.227911 24352 raft_consensus.cc:348] T 3f44a6a229f7456bafc8b24a837fb68a P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:54.228060 24352 raft_consensus.cc:374] T 3f44a6a229f7456bafc8b24a837fb68a P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:54.228094 24352 raft_consensus.cc:729] T 3f44a6a229f7456bafc8b24a837fb68a P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:54.228210 24352 consensus_queue.cc:260] T 3f44a6a229f7456bafc8b24a837fb68a P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:54.228806 24352 ts_tablet_manager.cc:1398] T 3f44a6a229f7456bafc8b24a837fb68a P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:54.228963 24418 tablet_bootstrap.cc:654] T 3f44a6a229f7456bafc8b24a837fb68a P 61b887f0d40d497badeca5604c1f0546: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:54.229648 24349 tablet_bootstrap.cc:492] T 3f44a6a229f7456bafc8b24a837fb68a P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:54.229737 24349 ts_tablet_manager.cc:1367] T 3f44a6a229f7456bafc8b24a837fb68a P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 13:56:54.230162 24349 raft_consensus.cc:348] T 3f44a6a229f7456bafc8b24a837fb68a P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:54.230263 24349 raft_consensus.cc:374] T 3f44a6a229f7456bafc8b24a837fb68a P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:54.230291 24349 raft_consensus.cc:729] T 3f44a6a229f7456bafc8b24a837fb68a P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:54.230406 24349 consensus_queue.cc:260] T 3f44a6a229f7456bafc8b24a837fb68a P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:54.230681 24349 ts_tablet_manager.cc:1398] T 3f44a6a229f7456bafc8b24a837fb68a P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:54.259250 24423 ts_tablet_manager.cc:902] T 30d2b5ede30744cfb46ec1be232a1405 P 009f849ee7b54c80a3f5264695047e4d: Initiating tablet copy from peer 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:54.262109 24423 tablet_copy_client.cc:308] T 30d2b5ede30744cfb46ec1be232a1405 P 009f849ee7b54c80a3f5264695047e4d: tablet copy: Beginning tablet copy session from remote peer at address 127.6.98.66:36387
I20240503 13:56:54.262624 13205 tablet_copy_service.cc:140] P 8ccf5a4ac450414c9d3579bd5800b770: Received BeginTabletCopySession request for tablet 30d2b5ede30744cfb46ec1be232a1405 from peer 009f849ee7b54c80a3f5264695047e4d ({username='slave'} at 127.6.98.70:57199)
I20240503 13:56:54.262718 13205 tablet_copy_service.cc:161] P 8ccf5a4ac450414c9d3579bd5800b770: Beginning new tablet copy session on tablet 30d2b5ede30744cfb46ec1be232a1405 from peer 009f849ee7b54c80a3f5264695047e4d at {username='slave'} at 127.6.98.70:57199: session id = 009f849ee7b54c80a3f5264695047e4d-30d2b5ede30744cfb46ec1be232a1405
I20240503 13:56:54.263515 13205 tablet_copy_source_session.cc:205] T 30d2b5ede30744cfb46ec1be232a1405 P 8ccf5a4ac450414c9d3579bd5800b770: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:56:54.264035 24412 tablet_bootstrap.cc:492] T df28831df92d4c12b63b2f7da6c6449a P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:56:54.264312 24412 tablet_bootstrap.cc:492] T df28831df92d4c12b63b2f7da6c6449a P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap complete.
I20240503 13:56:54.264520 24412 ts_tablet_manager.cc:1367] T df28831df92d4c12b63b2f7da6c6449a P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.047s user 0.024s sys 0.000s
I20240503 13:56:54.264945 24423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30d2b5ede30744cfb46ec1be232a1405. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:54.265144 24412 raft_consensus.cc:348] T df28831df92d4c12b63b2f7da6c6449a P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:54.265334 24412 raft_consensus.cc:729] T df28831df92d4c12b63b2f7da6c6449a P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:54.265540 24412 consensus_queue.cc:260] T df28831df92d4c12b63b2f7da6c6449a P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:54.267426 24423 tablet_copy_client.cc:783] T 30d2b5ede30744cfb46ec1be232a1405 P 009f849ee7b54c80a3f5264695047e4d: tablet copy: Starting download of 0 data blocks...
I20240503 13:56:54.267570 24423 tablet_copy_client.cc:647] T 30d2b5ede30744cfb46ec1be232a1405 P 009f849ee7b54c80a3f5264695047e4d: tablet copy: Starting download of 1 WAL segments...
I20240503 13:56:54.268834 24423 tablet_copy_client.cc:523] T 30d2b5ede30744cfb46ec1be232a1405 P 009f849ee7b54c80a3f5264695047e4d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:56:54.270092 24423 tablet_bootstrap.cc:492] T 30d2b5ede30744cfb46ec1be232a1405 P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:54.273077 24418 tablet_bootstrap.cc:492] T 3f44a6a229f7456bafc8b24a837fb68a P 61b887f0d40d497badeca5604c1f0546: No bootstrap required, opened a new log
I20240503 13:56:54.273178 24418 ts_tablet_manager.cc:1367] T 3f44a6a229f7456bafc8b24a837fb68a P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.046s user 0.002s sys 0.000s
I20240503 13:56:54.273813 24418 raft_consensus.cc:348] T 3f44a6a229f7456bafc8b24a837fb68a P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:54.273952 24418 raft_consensus.cc:374] T 3f44a6a229f7456bafc8b24a837fb68a P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:54.273984 24418 raft_consensus.cc:729] T 3f44a6a229f7456bafc8b24a837fb68a P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:54.274135 24418 consensus_queue.cc:260] T 3f44a6a229f7456bafc8b24a837fb68a P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:54.274416 24418 ts_tablet_manager.cc:1398] T 3f44a6a229f7456bafc8b24a837fb68a P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:54.277479 24412 ts_tablet_manager.cc:1398] T df28831df92d4c12b63b2f7da6c6449a P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.013s user 0.001s sys 0.000s
I20240503 13:56:54.278805 13205 tablet_copy_service.cc:342] P 8ccf5a4ac450414c9d3579bd5800b770: Request end of tablet copy session 76c386bef61a4bd5a907266a5118ffe2-df28831df92d4c12b63b2f7da6c6449a received from {username='slave'} at 127.6.98.71:41361
I20240503 13:56:54.278904 13205 tablet_copy_service.cc:433] P 8ccf5a4ac450414c9d3579bd5800b770: ending tablet copy session 76c386bef61a4bd5a907266a5118ffe2-df28831df92d4c12b63b2f7da6c6449a on tablet df28831df92d4c12b63b2f7da6c6449a with peer 76c386bef61a4bd5a907266a5118ffe2
I20240503 13:56:54.297366 24423 tablet_bootstrap.cc:492] T 30d2b5ede30744cfb46ec1be232a1405 P 009f849ee7b54c80a3f5264695047e4d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:56:54.297961 24423 tablet_bootstrap.cc:492] T 30d2b5ede30744cfb46ec1be232a1405 P 009f849ee7b54c80a3f5264695047e4d: Bootstrap complete.
I20240503 13:56:54.298343 24423 ts_tablet_manager.cc:1367] T 30d2b5ede30744cfb46ec1be232a1405 P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.028s user 0.020s sys 0.006s
I20240503 13:56:54.298894 24423 raft_consensus.cc:348] T 30d2b5ede30744cfb46ec1be232a1405 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:54.299086 24423 raft_consensus.cc:729] T 30d2b5ede30744cfb46ec1be232a1405 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:54.299263 24423 consensus_queue.cc:260] T 30d2b5ede30744cfb46ec1be232a1405 P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:54.299623 24423 ts_tablet_manager.cc:1398] T 30d2b5ede30744cfb46ec1be232a1405 P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:54.300207 13205 tablet_copy_service.cc:342] P 8ccf5a4ac450414c9d3579bd5800b770: Request end of tablet copy session 009f849ee7b54c80a3f5264695047e4d-30d2b5ede30744cfb46ec1be232a1405 received from {username='slave'} at 127.6.98.70:57199
I20240503 13:56:54.300305 13205 tablet_copy_service.cc:433] P 8ccf5a4ac450414c9d3579bd5800b770: ending tablet copy session 009f849ee7b54c80a3f5264695047e4d-30d2b5ede30744cfb46ec1be232a1405 on tablet 30d2b5ede30744cfb46ec1be232a1405 with peer 009f849ee7b54c80a3f5264695047e4d
I20240503 13:56:54.371556 24077 raft_consensus.cc:482] T 3f44a6a229f7456bafc8b24a837fb68a P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:54.371680 24077 raft_consensus.cc:504] T 3f44a6a229f7456bafc8b24a837fb68a P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:54.372251 24077 leader_election.cc:288] T 3f44a6a229f7456bafc8b24a837fb68a P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153), ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:54.373024 15492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f44a6a229f7456bafc8b24a837fb68a" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546" is_pre_election: true
I20240503 13:56:54.373183 15492 raft_consensus.cc:2454] T 3f44a6a229f7456bafc8b24a837fb68a P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 0.
I20240503 13:56:54.373467 13315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f44a6a229f7456bafc8b24a837fb68a" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703" is_pre_election: true
I20240503 13:56:54.373600 13315 raft_consensus.cc:2454] T 3f44a6a229f7456bafc8b24a837fb68a P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 0.
I20240503 13:56:54.374120 15561 leader_election.cc:302] T 3f44a6a229f7456bafc8b24a837fb68a P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:54.375177 24077 raft_consensus.cc:2781] T 3f44a6a229f7456bafc8b24a837fb68a P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:54.375241 24077 raft_consensus.cc:482] T 3f44a6a229f7456bafc8b24a837fb68a P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:54.375278 24077 raft_consensus.cc:3037] T 3f44a6a229f7456bafc8b24a837fb68a P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:54.376464 24077 raft_consensus.cc:504] T 3f44a6a229f7456bafc8b24a837fb68a P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:54.376797 24077 leader_election.cc:288] T 3f44a6a229f7456bafc8b24a837fb68a P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 election: Requested vote from peers 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153), ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:54.377409 15492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f44a6a229f7456bafc8b24a837fb68a" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546"
I20240503 13:56:54.377547 15492 raft_consensus.cc:3037] T 3f44a6a229f7456bafc8b24a837fb68a P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:54.377640 13315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f44a6a229f7456bafc8b24a837fb68a" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703"
I20240503 13:56:54.377749 13315 raft_consensus.cc:3037] T 3f44a6a229f7456bafc8b24a837fb68a P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:54.378567 15492 raft_consensus.cc:2454] T 3f44a6a229f7456bafc8b24a837fb68a P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 1.
I20240503 13:56:54.378712 13315 raft_consensus.cc:2454] T 3f44a6a229f7456bafc8b24a837fb68a P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 1.
I20240503 13:56:54.379630 15561 leader_election.cc:302] T 3f44a6a229f7456bafc8b24a837fb68a P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 61b887f0d40d497badeca5604c1f0546; no voters:
I20240503 13:56:54.379830 24077 raft_consensus.cc:2781] T 3f44a6a229f7456bafc8b24a837fb68a P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:54.379916 24077 raft_consensus.cc:686] T 3f44a6a229f7456bafc8b24a837fb68a P 009f849ee7b54c80a3f5264695047e4d [term 1 LEADER]: Becoming Leader. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Running, Role: LEADER
I20240503 13:56:54.380089 24077 consensus_queue.cc:237] T 3f44a6a229f7456bafc8b24a837fb68a P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:54.383342 12903 catalog_manager.cc:5484] T 3f44a6a229f7456bafc8b24a837fb68a P 009f849ee7b54c80a3f5264695047e4d reported cstate change: term changed from 0 to 1, leader changed from <none> to 009f849ee7b54c80a3f5264695047e4d (127.6.98.70). New cstate: current_term: 1 leader_uuid: "009f849ee7b54c80a3f5264695047e4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:54.399863 24369 consensus_queue.cc:1035] T e4e7d7a8644946a59bf5a3df7def80f4 P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:54.407905 23686 consensus_queue.cc:1035] T e4e7d7a8644946a59bf5a3df7def80f4 P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:54.412977 15604 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 13:56:54.425298 13034 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:56:54.428406 15473 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:56:54.448951 15867 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 13:56:54.449645 13163 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 13:56:54.450845 15738 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 13:56:54.461805 13427 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:54.474014 13295 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 13:56:54.478065 22256 consensus_queue.cc:1035] T dd39f3ab7cdb42b38267214056605328 P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:54.481335 23984 consensus_queue.cc:1035] T dd39f3ab7cdb42b38267214056605328 P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:54.509001 23843 consensus_queue.cc:1035] T 93d1e77b0d1d410093cb9233f03fa70e P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:54.513211 24235 consensus_queue.cc:1035] T 93d1e77b0d1d410093cb9233f03fa70e P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:54.524370 12903 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:56:54.528545 13315 raft_consensus.cc:1261] T 3f44a6a229f7456bafc8b24a837fb68a P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Refusing update from remote peer 009f849ee7b54c80a3f5264695047e4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:54.528793 15492 raft_consensus.cc:1261] T 3f44a6a229f7456bafc8b24a837fb68a P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Refusing update from remote peer 009f849ee7b54c80a3f5264695047e4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:54.529093 24077 consensus_queue.cc:1035] T 3f44a6a229f7456bafc8b24a837fb68a P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:54.529405 24077 consensus_queue.cc:1035] T 3f44a6a229f7456bafc8b24a837fb68a P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:54.531158 13184 consensus_queue.cc:237] T 7239f94a7c88433aa6062390a54e7562 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:54.532956 15623 raft_consensus.cc:1261] T 7239f94a7c88433aa6062390a54e7562 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Refusing update from remote peer 8ccf5a4ac450414c9d3579bd5800b770: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:56:54.533480 12903 catalog_manager.cc:5576] TS ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279) does not have the latest schema for tablet 3f44a6a229f7456bafc8b24a837fb68a (table rebalancer_dynamic_table [id=639667bd73754b769569bbc53d4e1d94]). Expected version 1 got 0
I20240503 13:56:54.533514 13445 raft_consensus.cc:1261] T 7239f94a7c88433aa6062390a54e7562 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer 8ccf5a4ac450414c9d3579bd5800b770: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:56:54.534480 12909 catalog_manager.cc:5576] TS 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153) does not have the latest schema for tablet 3f44a6a229f7456bafc8b24a837fb68a (table rebalancer_dynamic_table [id=639667bd73754b769569bbc53d4e1d94]). Expected version 1 got 0
I20240503 13:56:54.534968 24514 tablet.cc:1722] T 3f44a6a229f7456bafc8b24a837fb68a P ba0194098aa84449ae17e8a0d4c43703: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:54.535403 24514 tablet.cc:1620] T 3f44a6a229f7456bafc8b24a837fb68a P ba0194098aa84449ae17e8a0d4c43703: MemRowSet was empty: no flush needed.
I20240503 13:56:54.536013 24235 consensus_queue.cc:1035] T 7239f94a7c88433aa6062390a54e7562 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:54.536334 24235 consensus_queue.cc:1035] T 7239f94a7c88433aa6062390a54e7562 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:54.536692 12903 catalog_manager.cc:5576] TS ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279) does not have the latest schema for tablet 3f44a6a229f7456bafc8b24a837fb68a (table rebalancer_dynamic_table [id=639667bd73754b769569bbc53d4e1d94]). Expected version 1 got 0
I20240503 13:56:54.536000 24515 tablet.cc:1722] T 3f44a6a229f7456bafc8b24a837fb68a P 61b887f0d40d497badeca5604c1f0546: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:54.537267 24515 tablet.cc:1620] T 3f44a6a229f7456bafc8b24a837fb68a P 61b887f0d40d497badeca5604c1f0546: MemRowSet was empty: no flush needed.
I20240503 13:56:54.539626 24513 tablet.cc:1722] T 3f44a6a229f7456bafc8b24a837fb68a P 009f849ee7b54c80a3f5264695047e4d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:54.539935 24513 tablet.cc:1620] T 3f44a6a229f7456bafc8b24a837fb68a P 009f849ee7b54c80a3f5264695047e4d: MemRowSet was empty: no flush needed.
I20240503 13:56:54.541682 12903 catalog_manager.cc:5576] TS 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153) does not have the latest schema for tablet 3f44a6a229f7456bafc8b24a837fb68a (table rebalancer_dynamic_table [id=639667bd73754b769569bbc53d4e1d94]). Expected version 1 got 0
I20240503 13:56:54.541990 23843 raft_consensus.cc:2932] T 7239f94a7c88433aa6062390a54e7562 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:54.542371 15623 raft_consensus.cc:2932] T 7239f94a7c88433aa6062390a54e7562 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:54.543565 12909 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=639667bd73754b769569bbc53d4e1d94] alter complete (version 1)
I20240503 13:56:54.546487 12903 catalog_manager.cc:5484] T 7239f94a7c88433aa6062390a54e7562 P 009f849ee7b54c80a3f5264695047e4d reported cstate change: config changed from index 4 to 5, VOTER ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) evicted. New cstate: current_term: 1 leader_uuid: "8ccf5a4ac450414c9d3579bd5800b770" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:54.546535 13445 raft_consensus.cc:2932] T 7239f94a7c88433aa6062390a54e7562 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:54.551170 13295 tablet_service.cc:1514] Processing DeleteTablet for tablet 7239f94a7c88433aa6062390a54e7562 with delete_type TABLET_DATA_TOMBSTONED (TS ba0194098aa84449ae17e8a0d4c43703 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:33250
I20240503 13:56:54.551620 24523 tablet_replica.cc:330] T 7239f94a7c88433aa6062390a54e7562 P ba0194098aa84449ae17e8a0d4c43703: stopping tablet replica
I20240503 13:56:54.551820 24523 raft_consensus.cc:2229] T 7239f94a7c88433aa6062390a54e7562 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:54.552201 24523 raft_consensus.cc:2258] T 7239f94a7c88433aa6062390a54e7562 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:54.552784 24523 ts_tablet_manager.cc:1875] T 7239f94a7c88433aa6062390a54e7562 P ba0194098aa84449ae17e8a0d4c43703: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:56:54.554708 24523 ts_tablet_manager.cc:1888] T 7239f94a7c88433aa6062390a54e7562 P ba0194098aa84449ae17e8a0d4c43703: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 13:56:54.555301 24523 log.cc:1196] T 7239f94a7c88433aa6062390a54e7562 P ba0194098aa84449ae17e8a0d4c43703: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/wal/wals/7239f94a7c88433aa6062390a54e7562
I20240503 13:56:54.556100 12899 catalog_manager.cc:4830] TS ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279): tablet 7239f94a7c88433aa6062390a54e7562 (table rebalance_test_table_1 [id=ba90ff8253204f25829d57a56c0b5e94]) successfully deleted
I20240503 13:56:54.591358 15758 raft_consensus.cc:1203] T df28831df92d4c12b63b2f7da6c6449a P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:56:54.596442 13053 consensus_queue.cc:237] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:54.597796 13315 raft_consensus.cc:1261] T 7b57f64985b74e5fa75a5603ab8cc4bf P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Refusing update from remote peer 60ba3ffe3b7746469834cd91c84be14e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:56:54.598009 13445 raft_consensus.cc:1261] T 7b57f64985b74e5fa75a5603ab8cc4bf P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer 60ba3ffe3b7746469834cd91c84be14e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 13:56:54.598162 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7b57f64985b74e5fa75a5603ab8cc4bf. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:54.598482 24551 consensus_queue.cc:1035] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:54.599788 24551 consensus_queue.cc:1035] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:54.601411 24551 raft_consensus.cc:2932] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:54.601862 13315 raft_consensus.cc:2932] T 7b57f64985b74e5fa75a5603ab8cc4bf P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:54.604257 13445 raft_consensus.cc:2932] T 7b57f64985b74e5fa75a5603ab8cc4bf P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:54.604970 12908 catalog_manager.cc:5484] T 7b57f64985b74e5fa75a5603ab8cc4bf P ba0194098aa84449ae17e8a0d4c43703 reported cstate change: config changed from index -1 to 4, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) added. New cstate: current_term: 1 leader_uuid: "60ba3ffe3b7746469834cd91c84be14e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:54.646754 12903 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:56:54.653008 24515 tablet.cc:1722] T 3f44a6a229f7456bafc8b24a837fb68a P 61b887f0d40d497badeca5604c1f0546: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:54.652408 24513 tablet.cc:1722] T 3f44a6a229f7456bafc8b24a837fb68a P 009f849ee7b54c80a3f5264695047e4d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:54.653340 24515 tablet.cc:1620] T 3f44a6a229f7456bafc8b24a837fb68a P 61b887f0d40d497badeca5604c1f0546: MemRowSet was empty: no flush needed.
I20240503 13:56:54.653415 24513 tablet.cc:1620] T 3f44a6a229f7456bafc8b24a837fb68a P 009f849ee7b54c80a3f5264695047e4d: MemRowSet was empty: no flush needed.
I20240503 13:56:54.654271 24514 tablet.cc:1722] T 3f44a6a229f7456bafc8b24a837fb68a P ba0194098aa84449ae17e8a0d4c43703: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:54.654558 24514 tablet.cc:1620] T 3f44a6a229f7456bafc8b24a837fb68a P ba0194098aa84449ae17e8a0d4c43703: MemRowSet was empty: no flush needed.
I20240503 13:56:54.656045 12908 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_extra_table_66"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:56:54.661147 12909 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=639667bd73754b769569bbc53d4e1d94] alter complete (version 2)
I20240503 13:56:54.668848 13427 tablet_service.cc:1467] Processing CreateTablet for tablet 5edef9d909ac42d6b5acc22464a284c8 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=c53d76d8b5b1475dbab7cd20c6b7db35]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:54.669162 13427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5edef9d909ac42d6b5acc22464a284c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:54.669517 15868 tablet_service.cc:1467] Processing CreateTablet for tablet 5f34ac73ebe544a1b7a34e8c2d11568e (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=c53d76d8b5b1475dbab7cd20c6b7db35]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:54.669557 13424 tablet_service.cc:1467] Processing CreateTablet for tablet 5f34ac73ebe544a1b7a34e8c2d11568e (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=c53d76d8b5b1475dbab7cd20c6b7db35]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:54.669754 13424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f34ac73ebe544a1b7a34e8c2d11568e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:54.669806 15868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f34ac73ebe544a1b7a34e8c2d11568e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:54.671679 15869 tablet_service.cc:1467] Processing CreateTablet for tablet 7dca80c8a74a42c4b459315878ef5d14 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=c53d76d8b5b1475dbab7cd20c6b7db35]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:54.671854 13034 tablet_service.cc:1467] Processing CreateTablet for tablet 7dca80c8a74a42c4b459315878ef5d14 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=c53d76d8b5b1475dbab7cd20c6b7db35]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:54.669521 15867 tablet_service.cc:1467] Processing CreateTablet for tablet 5edef9d909ac42d6b5acc22464a284c8 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=c53d76d8b5b1475dbab7cd20c6b7db35]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:54.672116 13034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dca80c8a74a42c4b459315878ef5d14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:54.672240 15867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5edef9d909ac42d6b5acc22464a284c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:54.673198 15869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dca80c8a74a42c4b459315878ef5d14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:54.675004 24599 tablet_bootstrap.cc:492] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:54.675530 24597 tablet_bootstrap.cc:492] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:54.675606 13424 tablet_service.cc:1467] Processing CreateTablet for tablet 7dca80c8a74a42c4b459315878ef5d14 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=c53d76d8b5b1475dbab7cd20c6b7db35]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:54.675808 13424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dca80c8a74a42c4b459315878ef5d14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:54.676148 24599 tablet_bootstrap.cc:654] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:54.676330 24597 tablet_bootstrap.cc:654] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:54.677238 24599 tablet_bootstrap.cc:492] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:54.677318 24599 ts_tablet_manager.cc:1367] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:56:54.677804 24599 raft_consensus.cc:348] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:54.677908 24599 raft_consensus.cc:374] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:54.677939 24599 raft_consensus.cc:729] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:54.678064 24599 consensus_queue.cc:260] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:54.678352 24599 ts_tablet_manager.cc:1398] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:54.679687 15473 tablet_service.cc:1467] Processing CreateTablet for tablet 5f34ac73ebe544a1b7a34e8c2d11568e (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=c53d76d8b5b1475dbab7cd20c6b7db35]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:54.679689 15738 tablet_service.cc:1467] Processing CreateTablet for tablet 5edef9d909ac42d6b5acc22464a284c8 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=c53d76d8b5b1475dbab7cd20c6b7db35]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:54.679942 15473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f34ac73ebe544a1b7a34e8c2d11568e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:54.679963 15738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5edef9d909ac42d6b5acc22464a284c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:54.682026 24418 tablet_bootstrap.cc:492] T 5f34ac73ebe544a1b7a34e8c2d11568e P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:54.682915 24418 tablet_bootstrap.cc:654] T 5f34ac73ebe544a1b7a34e8c2d11568e P 61b887f0d40d497badeca5604c1f0546: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:54.684581 24602 tablet_bootstrap.cc:492] T 5edef9d909ac42d6b5acc22464a284c8 P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:54.685272 24418 tablet_bootstrap.cc:492] T 5f34ac73ebe544a1b7a34e8c2d11568e P 61b887f0d40d497badeca5604c1f0546: No bootstrap required, opened a new log
I20240503 13:56:54.685353 24418 ts_tablet_manager.cc:1367] T 5f34ac73ebe544a1b7a34e8c2d11568e P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:56:54.685492 24602 tablet_bootstrap.cc:654] T 5edef9d909ac42d6b5acc22464a284c8 P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:54.685793 24418 raft_consensus.cc:348] T 5f34ac73ebe544a1b7a34e8c2d11568e P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:54.685892 24418 raft_consensus.cc:374] T 5f34ac73ebe544a1b7a34e8c2d11568e P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:54.685920 24418 raft_consensus.cc:729] T 5f34ac73ebe544a1b7a34e8c2d11568e P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:54.686069 24418 consensus_queue.cc:260] T 5f34ac73ebe544a1b7a34e8c2d11568e P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:54.686326 24418 ts_tablet_manager.cc:1398] T 5f34ac73ebe544a1b7a34e8c2d11568e P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:54.688660 24598 tablet_bootstrap.cc:492] T 5f34ac73ebe544a1b7a34e8c2d11568e P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:54.689488 24602 tablet_bootstrap.cc:492] T 5edef9d909ac42d6b5acc22464a284c8 P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:54.689568 24598 tablet_bootstrap.cc:654] T 5f34ac73ebe544a1b7a34e8c2d11568e P e47105a0f46e4511b7112e571688a4c7: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:54.689813 24602 ts_tablet_manager.cc:1367] T 5edef9d909ac42d6b5acc22464a284c8 P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 13:56:54.690311 24602 raft_consensus.cc:348] T 5edef9d909ac42d6b5acc22464a284c8 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:54.690409 24602 raft_consensus.cc:374] T 5edef9d909ac42d6b5acc22464a284c8 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:54.690436 24602 raft_consensus.cc:729] T 5edef9d909ac42d6b5acc22464a284c8 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:54.690611 24602 consensus_queue.cc:260] T 5edef9d909ac42d6b5acc22464a284c8 P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:54.690683 24597 tablet_bootstrap.cc:492] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:54.690881 24602 ts_tablet_manager.cc:1398] T 5edef9d909ac42d6b5acc22464a284c8 P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:54.691045 24597 ts_tablet_manager.cc:1367] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.002s
I20240503 13:56:54.691581 24597 raft_consensus.cc:348] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:54.692447 24597 raft_consensus.cc:374] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:54.692554 24597 raft_consensus.cc:729] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:54.692739 24597 consensus_queue.cc:260] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:54.693152 24598 tablet_bootstrap.cc:492] T 5f34ac73ebe544a1b7a34e8c2d11568e P e47105a0f46e4511b7112e571688a4c7: No bootstrap required, opened a new log
I20240503 13:56:54.693295 24598 ts_tablet_manager.cc:1367] T 5f34ac73ebe544a1b7a34e8c2d11568e P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240503 13:56:54.693316 24597 ts_tablet_manager.cc:1398] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:54.693533 24597 tablet_bootstrap.cc:492] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:54.693800 24598 raft_consensus.cc:348] T 5f34ac73ebe544a1b7a34e8c2d11568e P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:54.693964 24598 raft_consensus.cc:374] T 5f34ac73ebe544a1b7a34e8c2d11568e P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:54.694337 24598 raft_consensus.cc:729] T 5f34ac73ebe544a1b7a34e8c2d11568e P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:54.694533 24598 consensus_queue.cc:260] T 5f34ac73ebe544a1b7a34e8c2d11568e P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:54.695067 24598 ts_tablet_manager.cc:1398] T 5f34ac73ebe544a1b7a34e8c2d11568e P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:56:54.695825 24598 tablet_bootstrap.cc:492] T 5edef9d909ac42d6b5acc22464a284c8 P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:54.695926 24597 tablet_bootstrap.cc:654] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:54.696997 24597 tablet_bootstrap.cc:492] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:54.697124 24597 ts_tablet_manager.cc:1367] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240503 13:56:54.697240 24598 tablet_bootstrap.cc:654] T 5edef9d909ac42d6b5acc22464a284c8 P e47105a0f46e4511b7112e571688a4c7: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:54.698725 24597 raft_consensus.cc:348] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:54.698892 24597 raft_consensus.cc:374] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:54.698984 24597 raft_consensus.cc:729] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:54.699160 24597 consensus_queue.cc:260] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:54.699496 24597 ts_tablet_manager.cc:1398] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 13:56:54.700038 24597 tablet_bootstrap.cc:492] T 7dca80c8a74a42c4b459315878ef5d14 P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:54.704033 24597 tablet_bootstrap.cc:654] T 7dca80c8a74a42c4b459315878ef5d14 P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:54.705219 15623 raft_consensus.cc:1203] T 30d2b5ede30744cfb46ec1be232a1405 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:56:54.708531 24597 tablet_bootstrap.cc:492] T 7dca80c8a74a42c4b459315878ef5d14 P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:54.708693 24597 ts_tablet_manager.cc:1367] T 7dca80c8a74a42c4b459315878ef5d14 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 13:56:54.709239 24597 raft_consensus.cc:348] T 7dca80c8a74a42c4b459315878ef5d14 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:54.709349 24597 raft_consensus.cc:374] T 7dca80c8a74a42c4b459315878ef5d14 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:54.709379 24597 raft_consensus.cc:729] T 7dca80c8a74a42c4b459315878ef5d14 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:54.709854 24597 consensus_queue.cc:260] T 7dca80c8a74a42c4b459315878ef5d14 P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:54.710249 24597 ts_tablet_manager.cc:1398] T 7dca80c8a74a42c4b459315878ef5d14 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:54.730232 24598 tablet_bootstrap.cc:492] T 5edef9d909ac42d6b5acc22464a284c8 P e47105a0f46e4511b7112e571688a4c7: No bootstrap required, opened a new log
I20240503 13:56:54.730352 24598 ts_tablet_manager.cc:1367] T 5edef9d909ac42d6b5acc22464a284c8 P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.035s user 0.002s sys 0.000s
I20240503 13:56:54.730821 24598 raft_consensus.cc:348] T 5edef9d909ac42d6b5acc22464a284c8 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:54.730927 24598 raft_consensus.cc:374] T 5edef9d909ac42d6b5acc22464a284c8 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:54.730955 24598 raft_consensus.cc:729] T 5edef9d909ac42d6b5acc22464a284c8 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:54.731076 24598 consensus_queue.cc:260] T 5edef9d909ac42d6b5acc22464a284c8 P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:54.731341 24598 ts_tablet_manager.cc:1398] T 5edef9d909ac42d6b5acc22464a284c8 P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:54.731490 24598 tablet_bootstrap.cc:492] T 7dca80c8a74a42c4b459315878ef5d14 P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:54.732414 24598 tablet_bootstrap.cc:654] T 7dca80c8a74a42c4b459315878ef5d14 P e47105a0f46e4511b7112e571688a4c7: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:54.747139 24598 tablet_bootstrap.cc:492] T 7dca80c8a74a42c4b459315878ef5d14 P e47105a0f46e4511b7112e571688a4c7: No bootstrap required, opened a new log
I20240503 13:56:54.747355 24598 ts_tablet_manager.cc:1367] T 7dca80c8a74a42c4b459315878ef5d14 P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240503 13:56:54.748078 24598 raft_consensus.cc:348] T 7dca80c8a74a42c4b459315878ef5d14 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:54.748180 24598 raft_consensus.cc:374] T 7dca80c8a74a42c4b459315878ef5d14 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:54.748208 24598 raft_consensus.cc:729] T 7dca80c8a74a42c4b459315878ef5d14 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:54.748317 24598 consensus_queue.cc:260] T 7dca80c8a74a42c4b459315878ef5d14 P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:54.749545 24598 ts_tablet_manager.cc:1398] T 7dca80c8a74a42c4b459315878ef5d14 P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:54.764746 12906 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:56:54.772449 24513 tablet.cc:1722] T 3f44a6a229f7456bafc8b24a837fb68a P 009f849ee7b54c80a3f5264695047e4d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:54.772925 24514 tablet.cc:1722] T 3f44a6a229f7456bafc8b24a837fb68a P ba0194098aa84449ae17e8a0d4c43703: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:54.776290 24515 tablet.cc:1722] T 3f44a6a229f7456bafc8b24a837fb68a P 61b887f0d40d497badeca5604c1f0546: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:54.780450 12909 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table_new_name [id=639667bd73754b769569bbc53d4e1d94] alter complete (version 3)
I20240503 13:56:54.796730 15604 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 13:56:54.801590 15473 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:56:54.807940 13424 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:56:54.817329 15869 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 13:56:54.819489 15738 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 13:56:54.820030 13163 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 13:56:54.828650 13295 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 13:56:54.842185 22256 raft_consensus.cc:482] T 5f34ac73ebe544a1b7a34e8c2d11568e P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:54.842332 22256 raft_consensus.cc:504] T 5f34ac73ebe544a1b7a34e8c2d11568e P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:54.842998 22256 leader_election.cc:288] T 5f34ac73ebe544a1b7a34e8c2d11568e P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153)
I20240503 13:56:54.843241 13034 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:56:54.845552 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f34ac73ebe544a1b7a34e8c2d11568e" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
I20240503 13:56:54.845556 15493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f34ac73ebe544a1b7a34e8c2d11568e" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546" is_pre_election: true
I20240503 13:56:54.845705 15493 raft_consensus.cc:2454] T 5f34ac73ebe544a1b7a34e8c2d11568e P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 0.
I20240503 13:56:54.845706 13445 raft_consensus.cc:2454] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 0.
I20240503 13:56:54.846014 15822 leader_election.cc:302] T 5f34ac73ebe544a1b7a34e8c2d11568e P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:54.846196 22256 raft_consensus.cc:2781] T 5f34ac73ebe544a1b7a34e8c2d11568e P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:54.846246 22256 raft_consensus.cc:482] T 5f34ac73ebe544a1b7a34e8c2d11568e P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:54.846300 22256 raft_consensus.cc:3037] T 5f34ac73ebe544a1b7a34e8c2d11568e P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:54.847512 22256 raft_consensus.cc:504] T 5f34ac73ebe544a1b7a34e8c2d11568e P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:54.847893 22256 leader_election.cc:288] T 5f34ac73ebe544a1b7a34e8c2d11568e P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 election: Requested vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153)
I20240503 13:56:54.848798 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f34ac73ebe544a1b7a34e8c2d11568e" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
I20240503 13:56:54.848909 13445 raft_consensus.cc:3037] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:54.852675 15493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f34ac73ebe544a1b7a34e8c2d11568e" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546"
I20240503 13:56:54.852814 15493 raft_consensus.cc:3037] T 5f34ac73ebe544a1b7a34e8c2d11568e P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:54.853869 15493 raft_consensus.cc:2454] T 5f34ac73ebe544a1b7a34e8c2d11568e P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 1.
I20240503 13:56:54.857985 15825 leader_election.cc:302] T 5f34ac73ebe544a1b7a34e8c2d11568e P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61b887f0d40d497badeca5604c1f0546, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:54.858451 13445 raft_consensus.cc:2454] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 1.
I20240503 13:56:54.858810 23984 raft_consensus.cc:2781] T 5f34ac73ebe544a1b7a34e8c2d11568e P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:54.859041 23984 raft_consensus.cc:686] T 5f34ac73ebe544a1b7a34e8c2d11568e P e47105a0f46e4511b7112e571688a4c7 [term 1 LEADER]: Becoming Leader. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Running, Role: LEADER
I20240503 13:56:54.859184 23984 consensus_queue.cc:237] T 5f34ac73ebe544a1b7a34e8c2d11568e P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:54.870589 12906 catalog_manager.cc:5484] T 5f34ac73ebe544a1b7a34e8c2d11568e P e47105a0f46e4511b7112e571688a4c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e47105a0f46e4511b7112e571688a4c7 (127.6.98.72). New cstate: current_term: 1 leader_uuid: "e47105a0f46e4511b7112e571688a4c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:54.886044 23971 raft_consensus.cc:482] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:54.886174 23971 raft_consensus.cc:504] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:54.886708 23971 leader_election.cc:288] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:54.887006 15888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5edef9d909ac42d6b5acc22464a284c8" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7" is_pre_election: true
I20240503 13:56:54.887143 15888 raft_consensus.cc:2454] T 5edef9d909ac42d6b5acc22464a284c8 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 0.
I20240503 13:56:54.887193 12906 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:54.887225 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5edef9d909ac42d6b5acc22464a284c8" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2" is_pre_election: true
I20240503 13:56:54.887319 12906 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:54.887364 15757 raft_consensus.cc:2454] T 5edef9d909ac42d6b5acc22464a284c8 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 0.
I20240503 13:56:54.887410 13380 leader_election.cc:302] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:54.887952 23971 raft_consensus.cc:2781] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:54.888024 23971 raft_consensus.cc:482] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:54.888056 23971 raft_consensus.cc:3037] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:54.889170 23971 raft_consensus.cc:504] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:54.889549 23971 leader_election.cc:288] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 election: Requested vote from peers 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:54.890352 12906 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Sending DeleteTablet for 3 replicas of tablet 3f44a6a229f7456bafc8b24a837fb68a
I20240503 13:56:54.890328 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5edef9d909ac42d6b5acc22464a284c8" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2"
I20240503 13:56:54.890444 15757 raft_consensus.cc:3037] T 5edef9d909ac42d6b5acc22464a284c8 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:54.891049 15604 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f44a6a229f7456bafc8b24a837fb68a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:54 UTC) from {username='slave'} at 127.0.0.1:60718
I20240503 13:56:54.891044 13295 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f44a6a229f7456bafc8b24a837fb68a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:54 UTC) from {username='slave'} at 127.0.0.1:33250
I20240503 13:56:54.891227 24523 tablet_replica.cc:330] T 3f44a6a229f7456bafc8b24a837fb68a P ba0194098aa84449ae17e8a0d4c43703: stopping tablet replica
I20240503 13:56:54.891358 24523 raft_consensus.cc:2229] T 3f44a6a229f7456bafc8b24a837fb68a P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:54.891357 15888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5edef9d909ac42d6b5acc22464a284c8" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47105a0f46e4511b7112e571688a4c7"
I20240503 13:56:54.891459 24640 tablet_replica.cc:330] T 3f44a6a229f7456bafc8b24a837fb68a P 009f849ee7b54c80a3f5264695047e4d: stopping tablet replica
I20240503 13:56:54.891482 15888 raft_consensus.cc:3037] T 5edef9d909ac42d6b5acc22464a284c8 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:54.891487 24523 raft_consensus.cc:2258] T 3f44a6a229f7456bafc8b24a837fb68a P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:54.891579 15757 raft_consensus.cc:2454] T 5edef9d909ac42d6b5acc22464a284c8 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 1.
I20240503 13:56:54.891587 24640 raft_consensus.cc:2229] T 3f44a6a229f7456bafc8b24a837fb68a P 009f849ee7b54c80a3f5264695047e4d [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:56:54.891774 24640 raft_consensus.cc:2258] T 3f44a6a229f7456bafc8b24a837fb68a P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:54.891855 13383 leader_election.cc:302] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, 76c386bef61a4bd5a907266a5118ffe2; no voters:
I20240503 13:56:54.892293 24523 ts_tablet_manager.cc:1875] T 3f44a6a229f7456bafc8b24a837fb68a P ba0194098aa84449ae17e8a0d4c43703: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:54.892293 24640 ts_tablet_manager.cc:1875] T 3f44a6a229f7456bafc8b24a837fb68a P 009f849ee7b54c80a3f5264695047e4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:54.892525 15888 raft_consensus.cc:2454] T 5edef9d909ac42d6b5acc22464a284c8 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 1.
I20240503 13:56:54.893550 23971 raft_consensus.cc:2781] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:54.893635 23971 raft_consensus.cc:686] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 LEADER]: Becoming Leader. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Running, Role: LEADER
I20240503 13:56:54.893769 23971 consensus_queue.cc:237] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:54.894496 24523 ts_tablet_manager.cc:1888] T 3f44a6a229f7456bafc8b24a837fb68a P ba0194098aa84449ae17e8a0d4c43703: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:54.894578 24523 log.cc:1196] T 3f44a6a229f7456bafc8b24a837fb68a P ba0194098aa84449ae17e8a0d4c43703: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/wal/wals/3f44a6a229f7456bafc8b24a837fb68a
I20240503 13:56:54.894498 24640 ts_tablet_manager.cc:1888] T 3f44a6a229f7456bafc8b24a837fb68a P 009f849ee7b54c80a3f5264695047e4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:54.894629 24640 log.cc:1196] T 3f44a6a229f7456bafc8b24a837fb68a P 009f849ee7b54c80a3f5264695047e4d: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/wal/wals/3f44a6a229f7456bafc8b24a837fb68a
I20240503 13:56:54.894824 24523 ts_tablet_manager.cc:1909] T 3f44a6a229f7456bafc8b24a837fb68a P ba0194098aa84449ae17e8a0d4c43703: Deleting consensus metadata
I20240503 13:56:54.894845 24640 ts_tablet_manager.cc:1909] T 3f44a6a229f7456bafc8b24a837fb68a P 009f849ee7b54c80a3f5264695047e4d: Deleting consensus metadata
I20240503 13:56:54.895030 15473 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f44a6a229f7456bafc8b24a837fb68a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:54 UTC) from {username='slave'} at 127.0.0.1:58758
I20240503 13:56:54.895445 24645 tablet_replica.cc:330] T 3f44a6a229f7456bafc8b24a837fb68a P 61b887f0d40d497badeca5604c1f0546: stopping tablet replica
I20240503 13:56:54.895634 24645 raft_consensus.cc:2229] T 3f44a6a229f7456bafc8b24a837fb68a P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:54.895797 24645 raft_consensus.cc:2258] T 3f44a6a229f7456bafc8b24a837fb68a P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:54.896343 12899 catalog_manager.cc:4830] TS ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279): tablet 3f44a6a229f7456bafc8b24a837fb68a (table rebalancer_dynamic_table_new_name [id=639667bd73754b769569bbc53d4e1d94]) successfully deleted
I20240503 13:56:54.896713 24645 ts_tablet_manager.cc:1875] T 3f44a6a229f7456bafc8b24a837fb68a P 61b887f0d40d497badeca5604c1f0546: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:54.896937 12896 catalog_manager.cc:4830] TS 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951): tablet 3f44a6a229f7456bafc8b24a837fb68a (table rebalancer_dynamic_table_new_name [id=639667bd73754b769569bbc53d4e1d94]) successfully deleted
I20240503 13:56:54.897282 12906 catalog_manager.cc:5484] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68). New cstate: current_term: 1 leader_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:54.899032 24645 ts_tablet_manager.cc:1888] T 3f44a6a229f7456bafc8b24a837fb68a P 61b887f0d40d497badeca5604c1f0546: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:54.899111 24645 log.cc:1196] T 3f44a6a229f7456bafc8b24a837fb68a P 61b887f0d40d497badeca5604c1f0546: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/wal/wals/3f44a6a229f7456bafc8b24a837fb68a
I20240503 13:56:54.899391 24645 ts_tablet_manager.cc:1909] T 3f44a6a229f7456bafc8b24a837fb68a P 61b887f0d40d497badeca5604c1f0546: Deleting consensus metadata
I20240503 13:56:54.900265 12899 catalog_manager.cc:4830] TS 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153): tablet 3f44a6a229f7456bafc8b24a837fb68a (table rebalancer_dynamic_table_new_name [id=639667bd73754b769569bbc53d4e1d94]) successfully deleted
I20240503 13:56:54.927850 24638 raft_consensus.cc:482] T 7dca80c8a74a42c4b459315878ef5d14 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:54.928026 24638 raft_consensus.cc:504] T 7dca80c8a74a42c4b459315878ef5d14 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:54.928617 24638 leader_election.cc:288] T 7dca80c8a74a42c4b459315878ef5d14 P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:54.928898 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dca80c8a74a42c4b459315878ef5d14" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
I20240503 13:56:54.929050 13445 raft_consensus.cc:2454] T 7dca80c8a74a42c4b459315878ef5d14 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 0.
I20240503 13:56:54.929216 13053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dca80c8a74a42c4b459315878ef5d14" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e" is_pre_election: true
I20240503 13:56:54.929374 13053 raft_consensus.cc:2454] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 0.
I20240503 13:56:54.929695 15823 leader_election.cc:302] T 7dca80c8a74a42c4b459315878ef5d14 P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:54.930140 22256 raft_consensus.cc:2781] T 7dca80c8a74a42c4b459315878ef5d14 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:54.930195 22256 raft_consensus.cc:482] T 7dca80c8a74a42c4b459315878ef5d14 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:54.930231 22256 raft_consensus.cc:3037] T 7dca80c8a74a42c4b459315878ef5d14 P e47105a0f46e4511b7112e571688a4c7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:54.931488 22256 raft_consensus.cc:504] T 7dca80c8a74a42c4b459315878ef5d14 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:54.932612 22256 leader_election.cc:288] T 7dca80c8a74a42c4b459315878ef5d14 P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 election: Requested vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:54.933053 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dca80c8a74a42c4b459315878ef5d14" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
I20240503 13:56:54.933189 13445 raft_consensus.cc:3037] T 7dca80c8a74a42c4b459315878ef5d14 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:54.934370 13445 raft_consensus.cc:2454] T 7dca80c8a74a42c4b459315878ef5d14 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 1.
I20240503 13:56:54.934741 15822 leader_election.cc:302] T 7dca80c8a74a42c4b459315878ef5d14 P e47105a0f46e4511b7112e571688a4c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, e47105a0f46e4511b7112e571688a4c7; no voters:
I20240503 13:56:54.935065 22256 raft_consensus.cc:2781] T 7dca80c8a74a42c4b459315878ef5d14 P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:54.935007 13053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dca80c8a74a42c4b459315878ef5d14" candidate_uuid: "e47105a0f46e4511b7112e571688a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e"
I20240503 13:56:54.935148 22256 raft_consensus.cc:686] T 7dca80c8a74a42c4b459315878ef5d14 P e47105a0f46e4511b7112e571688a4c7 [term 1 LEADER]: Becoming Leader. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Running, Role: LEADER
I20240503 13:56:54.935164 13053 raft_consensus.cc:3037] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:54.935283 22256 consensus_queue.cc:237] T 7dca80c8a74a42c4b459315878ef5d14 P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:54.936318 13053 raft_consensus.cc:2454] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e47105a0f46e4511b7112e571688a4c7 in term 1.
I20240503 13:56:54.938215 12906 catalog_manager.cc:5484] T 7dca80c8a74a42c4b459315878ef5d14 P e47105a0f46e4511b7112e571688a4c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e47105a0f46e4511b7112e571688a4c7 (127.6.98.72). New cstate: current_term: 1 leader_uuid: "e47105a0f46e4511b7112e571688a4c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } }
I20240503 13:56:54.991997 12905 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:56:55.001332 13163 tablet_service.cc:1467] Processing CreateTablet for tablet 3fd497110cdb46fa9b51dd18e5408a5e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=089c57df50994be18a8170195960a2f5]), partition=
I20240503 13:56:55.001402 13034 tablet_service.cc:1467] Processing CreateTablet for tablet 3fd497110cdb46fa9b51dd18e5408a5e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=089c57df50994be18a8170195960a2f5]), partition=
I20240503 13:56:55.001448 13295 tablet_service.cc:1467] Processing CreateTablet for tablet 3fd497110cdb46fa9b51dd18e5408a5e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=089c57df50994be18a8170195960a2f5]), partition=
I20240503 13:56:55.001658 13163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fd497110cdb46fa9b51dd18e5408a5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:55.001658 13034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fd497110cdb46fa9b51dd18e5408a5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:55.001752 13295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fd497110cdb46fa9b51dd18e5408a5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:55.004065 24599 tablet_bootstrap.cc:492] T 3fd497110cdb46fa9b51dd18e5408a5e P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:55.005151 24599 tablet_bootstrap.cc:654] T 3fd497110cdb46fa9b51dd18e5408a5e P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:55.007108 24599 tablet_bootstrap.cc:492] T 3fd497110cdb46fa9b51dd18e5408a5e P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:55.007206 24599 ts_tablet_manager.cc:1367] T 3fd497110cdb46fa9b51dd18e5408a5e P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:56:55.007700 24599 raft_consensus.cc:348] T 3fd497110cdb46fa9b51dd18e5408a5e P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:55.007815 24599 raft_consensus.cc:374] T 3fd497110cdb46fa9b51dd18e5408a5e P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:55.007843 24599 raft_consensus.cc:729] T 3fd497110cdb46fa9b51dd18e5408a5e P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:55.007959 24599 consensus_queue.cc:260] T 3fd497110cdb46fa9b51dd18e5408a5e P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:55.008250 24599 ts_tablet_manager.cc:1398] T 3fd497110cdb46fa9b51dd18e5408a5e P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:55.009055 24687 tablet_bootstrap.cc:492] T 3fd497110cdb46fa9b51dd18e5408a5e P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:55.010066 24687 tablet_bootstrap.cc:654] T 3fd497110cdb46fa9b51dd18e5408a5e P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:55.022903 24686 tablet_bootstrap.cc:492] T 3fd497110cdb46fa9b51dd18e5408a5e P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:55.024596 24686 tablet_bootstrap.cc:654] T 3fd497110cdb46fa9b51dd18e5408a5e P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:55.026558 24687 tablet_bootstrap.cc:492] T 3fd497110cdb46fa9b51dd18e5408a5e P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:55.026669 24686 tablet_bootstrap.cc:492] T 3fd497110cdb46fa9b51dd18e5408a5e P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:55.026687 24687 ts_tablet_manager.cc:1367] T 3fd497110cdb46fa9b51dd18e5408a5e P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20240503 13:56:55.026751 24686 ts_tablet_manager.cc:1367] T 3fd497110cdb46fa9b51dd18e5408a5e P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20240503 13:56:55.027184 24687 raft_consensus.cc:348] T 3fd497110cdb46fa9b51dd18e5408a5e P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:55.027316 24687 raft_consensus.cc:374] T 3fd497110cdb46fa9b51dd18e5408a5e P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:55.027345 24687 raft_consensus.cc:729] T 3fd497110cdb46fa9b51dd18e5408a5e P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:55.027463 24687 consensus_queue.cc:260] T 3fd497110cdb46fa9b51dd18e5408a5e P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:55.027769 24687 ts_tablet_manager.cc:1398] T 3fd497110cdb46fa9b51dd18e5408a5e P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:55.027216 24686 raft_consensus.cc:348] T 3fd497110cdb46fa9b51dd18e5408a5e P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:55.031548 24686 raft_consensus.cc:374] T 3fd497110cdb46fa9b51dd18e5408a5e P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:55.031625 24686 raft_consensus.cc:729] T 3fd497110cdb46fa9b51dd18e5408a5e P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:55.031775 24686 consensus_queue.cc:260] T 3fd497110cdb46fa9b51dd18e5408a5e P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:55.032142 24686 ts_tablet_manager.cc:1398] T 3fd497110cdb46fa9b51dd18e5408a5e P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20240503 13:56:55.092191 13034 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:56:55.096123 15604 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 13:56:55.108374 15473 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:56:55.112226 13424 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 13:56:55.129688 15869 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 13:56:55.151593 13163 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 13:56:55.157305 13295 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 13:56:55.162012 24703 ts_tablet_manager.cc:902] T 7b57f64985b74e5fa75a5603ab8cc4bf P e47105a0f46e4511b7112e571688a4c7: Initiating tablet copy from peer 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:55.163369 24703 tablet_copy_client.cc:308] T 7b57f64985b74e5fa75a5603ab8cc4bf P e47105a0f46e4511b7112e571688a4c7: tablet copy: Beginning tablet copy session from remote peer at address 127.6.98.65:34695
I20240503 13:56:55.164420 13074 tablet_copy_service.cc:140] P 60ba3ffe3b7746469834cd91c84be14e: Received BeginTabletCopySession request for tablet 7b57f64985b74e5fa75a5603ab8cc4bf from peer e47105a0f46e4511b7112e571688a4c7 ({username='slave'} at 127.6.98.72:59759)
I20240503 13:56:55.164548 13074 tablet_copy_service.cc:161] P 60ba3ffe3b7746469834cd91c84be14e: Beginning new tablet copy session on tablet 7b57f64985b74e5fa75a5603ab8cc4bf from peer e47105a0f46e4511b7112e571688a4c7 at {username='slave'} at 127.6.98.72:59759: session id = e47105a0f46e4511b7112e571688a4c7-7b57f64985b74e5fa75a5603ab8cc4bf
I20240503 13:56:55.165660 13074 tablet_copy_source_session.cc:205] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:56:55.182494 15738 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 13:56:55.202751 24703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b57f64985b74e5fa75a5603ab8cc4bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:55.210937 24703 tablet_copy_client.cc:783] T 7b57f64985b74e5fa75a5603ab8cc4bf P e47105a0f46e4511b7112e571688a4c7: tablet copy: Starting download of 0 data blocks...
I20240503 13:56:55.211115 24703 tablet_copy_client.cc:647] T 7b57f64985b74e5fa75a5603ab8cc4bf P e47105a0f46e4511b7112e571688a4c7: tablet copy: Starting download of 1 WAL segments...
I20240503 13:56:55.212285 24703 tablet_copy_client.cc:523] T 7b57f64985b74e5fa75a5603ab8cc4bf P e47105a0f46e4511b7112e571688a4c7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:56:55.213833 24703 tablet_bootstrap.cc:492] T 7b57f64985b74e5fa75a5603ab8cc4bf P e47105a0f46e4511b7112e571688a4c7: Bootstrap starting.
I20240503 13:56:55.257186 24703 tablet_bootstrap.cc:492] T 7b57f64985b74e5fa75a5603ab8cc4bf P e47105a0f46e4511b7112e571688a4c7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:56:55.257519 24703 tablet_bootstrap.cc:492] T 7b57f64985b74e5fa75a5603ab8cc4bf P e47105a0f46e4511b7112e571688a4c7: Bootstrap complete.
I20240503 13:56:55.257728 24703 ts_tablet_manager.cc:1367] T 7b57f64985b74e5fa75a5603ab8cc4bf P e47105a0f46e4511b7112e571688a4c7: Time spent bootstrapping tablet: real 0.044s user 0.019s sys 0.003s
I20240503 13:56:55.258209 24703 raft_consensus.cc:348] T 7b57f64985b74e5fa75a5603ab8cc4bf P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:55.258342 24703 raft_consensus.cc:729] T 7b57f64985b74e5fa75a5603ab8cc4bf P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47105a0f46e4511b7112e571688a4c7, State: Initialized, Role: FOLLOWER
I20240503 13:56:55.258529 24703 consensus_queue.cc:260] T 7b57f64985b74e5fa75a5603ab8cc4bf P e47105a0f46e4511b7112e571688a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:55.258844 24703 ts_tablet_manager.cc:1398] T 7b57f64985b74e5fa75a5603ab8cc4bf P e47105a0f46e4511b7112e571688a4c7: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 13:56:55.259441 13074 tablet_copy_service.cc:342] P 60ba3ffe3b7746469834cd91c84be14e: Request end of tablet copy session e47105a0f46e4511b7112e571688a4c7-7b57f64985b74e5fa75a5603ab8cc4bf received from {username='slave'} at 127.6.98.72:59759
I20240503 13:56:55.259526 13074 tablet_copy_service.cc:433] P 60ba3ffe3b7746469834cd91c84be14e: ending tablet copy session e47105a0f46e4511b7112e571688a4c7-7b57f64985b74e5fa75a5603ab8cc4bf on tablet 7b57f64985b74e5fa75a5603ab8cc4bf with peer e47105a0f46e4511b7112e571688a4c7
I20240503 13:56:55.265609 13184 consensus_queue.cc:237] T 30d2b5ede30744cfb46ec1be232a1405 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:55.266976 15623 raft_consensus.cc:1261] T 30d2b5ede30744cfb46ec1be232a1405 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Refusing update from remote peer 8ccf5a4ac450414c9d3579bd5800b770: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:56:55.266956 13053 raft_consensus.cc:1261] T 30d2b5ede30744cfb46ec1be232a1405 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Refusing update from remote peer 8ccf5a4ac450414c9d3579bd5800b770: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:56:55.267428 24518 consensus_queue.cc:1035] T 30d2b5ede30744cfb46ec1be232a1405 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:55.267599 24235 consensus_queue.cc:1035] T 30d2b5ede30744cfb46ec1be232a1405 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:55.269392 24235 raft_consensus.cc:2932] T 30d2b5ede30744cfb46ec1be232a1405 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:55.269994 15623 raft_consensus.cc:2932] T 30d2b5ede30744cfb46ec1be232a1405 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:55.273694 12907 catalog_manager.cc:5484] T 30d2b5ede30744cfb46ec1be232a1405 P 009f849ee7b54c80a3f5264695047e4d reported cstate change: config changed from index 4 to 5, VOTER ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) evicted. New cstate: current_term: 1 leader_uuid: "8ccf5a4ac450414c9d3579bd5800b770" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:55.274852 24059 raft_consensus.cc:482] T 3fd497110cdb46fa9b51dd18e5408a5e P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:55.274950 24059 raft_consensus.cc:504] T 3fd497110cdb46fa9b51dd18e5408a5e P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:55.275357 24059 leader_election.cc:288] T 3fd497110cdb46fa9b51dd18e5408a5e P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:55.275936 13051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd497110cdb46fa9b51dd18e5408a5e" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e" is_pre_election: true
I20240503 13:56:55.276079 13051 raft_consensus.cc:2454] T 3fd497110cdb46fa9b51dd18e5408a5e P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 0.
I20240503 13:56:55.276400 13250 leader_election.cc:302] T 3fd497110cdb46fa9b51dd18e5408a5e P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:55.276587 13184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd497110cdb46fa9b51dd18e5408a5e" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770" is_pre_election: true
I20240503 13:56:55.276715 13184 raft_consensus.cc:2454] T 3fd497110cdb46fa9b51dd18e5408a5e P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 0.
I20240503 13:56:55.277010 24059 raft_consensus.cc:2781] T 3fd497110cdb46fa9b51dd18e5408a5e P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:55.277074 24059 raft_consensus.cc:482] T 3fd497110cdb46fa9b51dd18e5408a5e P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:55.277112 24059 raft_consensus.cc:3037] T 3fd497110cdb46fa9b51dd18e5408a5e P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:55.277652 13053 raft_consensus.cc:2932] T 30d2b5ede30744cfb46ec1be232a1405 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:55.278118 24059 raft_consensus.cc:504] T 3fd497110cdb46fa9b51dd18e5408a5e P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:55.278439 24059 leader_election.cc:288] T 3fd497110cdb46fa9b51dd18e5408a5e P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: Requested vote from peers 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:55.279495 13295 tablet_service.cc:1514] Processing DeleteTablet for tablet 30d2b5ede30744cfb46ec1be232a1405 with delete_type TABLET_DATA_TOMBSTONED (TS ba0194098aa84449ae17e8a0d4c43703 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:33250
I20240503 13:56:55.279529 13184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd497110cdb46fa9b51dd18e5408a5e" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770"
I20240503 13:56:55.279620 13184 raft_consensus.cc:3037] T 3fd497110cdb46fa9b51dd18e5408a5e P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:55.279722 24523 tablet_replica.cc:330] T 30d2b5ede30744cfb46ec1be232a1405 P ba0194098aa84449ae17e8a0d4c43703: stopping tablet replica
I20240503 13:56:55.279861 24523 raft_consensus.cc:2229] T 30d2b5ede30744cfb46ec1be232a1405 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:55.280021 24523 raft_consensus.cc:2258] T 30d2b5ede30744cfb46ec1be232a1405 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:55.280167 13051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd497110cdb46fa9b51dd18e5408a5e" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e"
I20240503 13:56:55.280287 13051 raft_consensus.cc:3037] T 3fd497110cdb46fa9b51dd18e5408a5e P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:55.280567 13184 raft_consensus.cc:2454] T 3fd497110cdb46fa9b51dd18e5408a5e P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 1.
I20240503 13:56:55.280659 24523 ts_tablet_manager.cc:1875] T 30d2b5ede30744cfb46ec1be232a1405 P ba0194098aa84449ae17e8a0d4c43703: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:56:55.280872 13252 leader_election.cc:302] T 3fd497110cdb46fa9b51dd18e5408a5e P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ccf5a4ac450414c9d3579bd5800b770, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:55.280835 23971 consensus_queue.cc:1035] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:55.281549 13051 raft_consensus.cc:2454] T 3fd497110cdb46fa9b51dd18e5408a5e P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 1.
I20240503 13:56:55.281566 24059 raft_consensus.cc:2781] T 3fd497110cdb46fa9b51dd18e5408a5e P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:55.281641 24059 raft_consensus.cc:686] T 3fd497110cdb46fa9b51dd18e5408a5e P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Becoming Leader. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Running, Role: LEADER
I20240503 13:56:55.281754 24059 consensus_queue.cc:237] T 3fd497110cdb46fa9b51dd18e5408a5e P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:55.284099 24637 consensus_queue.cc:1035] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:55.284754 12910 catalog_manager.cc:5484] T 3fd497110cdb46fa9b51dd18e5408a5e P ba0194098aa84449ae17e8a0d4c43703 reported cstate change: term changed from 0 to 1, leader changed from <none> to ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67). New cstate: current_term: 1 leader_uuid: "ba0194098aa84449ae17e8a0d4c43703" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:55.284915 24523 ts_tablet_manager.cc:1888] T 30d2b5ede30744cfb46ec1be232a1405 P ba0194098aa84449ae17e8a0d4c43703: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 13:56:55.284997 24523 log.cc:1196] T 30d2b5ede30744cfb46ec1be232a1405 P ba0194098aa84449ae17e8a0d4c43703: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/wal/wals/30d2b5ede30744cfb46ec1be232a1405
I20240503 13:56:55.287564 12899 catalog_manager.cc:4830] TS ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279): tablet 30d2b5ede30744cfb46ec1be232a1405 (table rebalance_test_table_7 [id=5e6f2f3703e1424ab61d8fe49e6266dd]) successfully deleted
I20240503 13:56:55.314669 24638 consensus_queue.cc:1035] T 7dca80c8a74a42c4b459315878ef5d14 P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:55.325656 23984 consensus_queue.cc:1035] T 7dca80c8a74a42c4b459315878ef5d14 P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:55.409893 23984 consensus_queue.cc:1035] T 5f34ac73ebe544a1b7a34e8c2d11568e P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:55.429594 24638 consensus_queue.cc:1035] T 5f34ac73ebe544a1b7a34e8c2d11568e P e47105a0f46e4511b7112e571688a4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:55.438647 13034 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:56:55.442414 15604 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 13:56:55.446676 15869 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 13:56:55.449252 12907 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:56:55.452718 15473 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:56:55.458447 13424 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 13:56:55.479254 13163 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 13:56:55.481750 12907 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_extra_table_68"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:56:55.492877 13295 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 13:56:55.493218 15738 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 13:56:55.498314 13424 tablet_service.cc:1467] Processing CreateTablet for tablet 5140b9d0611646fda51207df60c8a4df (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=7b7bd4cfcff14b02b707aa8303aece3a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:55.498662 13424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5140b9d0611646fda51207df60c8a4df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:55.499056 13296 tablet_service.cc:1467] Processing CreateTablet for tablet 5140b9d0611646fda51207df60c8a4df (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=7b7bd4cfcff14b02b707aa8303aece3a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:55.499329 13296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5140b9d0611646fda51207df60c8a4df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:55.499441 15738 tablet_service.cc:1467] Processing CreateTablet for tablet 96b5182e1e64427eae541e3c0a78b20a (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=7b7bd4cfcff14b02b707aa8303aece3a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:55.499684 15738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96b5182e1e64427eae541e3c0a78b20a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:55.501013 15604 tablet_service.cc:1467] Processing CreateTablet for tablet 26f3c461ddfd4e30b2485ab0386c89b2 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=7b7bd4cfcff14b02b707aa8303aece3a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:55.501276 15604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26f3c461ddfd4e30b2485ab0386c89b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:55.501243 13295 tablet_service.cc:1467] Processing CreateTablet for tablet 96b5182e1e64427eae541e3c0a78b20a (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=7b7bd4cfcff14b02b707aa8303aece3a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:55.501444 13295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96b5182e1e64427eae541e3c0a78b20a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:55.501578 24687 tablet_bootstrap.cc:492] T 5140b9d0611646fda51207df60c8a4df P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:55.502429 24687 tablet_bootstrap.cc:654] T 5140b9d0611646fda51207df60c8a4df P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:55.503636 15473 tablet_service.cc:1467] Processing CreateTablet for tablet 26f3c461ddfd4e30b2485ab0386c89b2 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=7b7bd4cfcff14b02b707aa8303aece3a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:55.503654 13165 tablet_service.cc:1467] Processing CreateTablet for tablet 26f3c461ddfd4e30b2485ab0386c89b2 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=7b7bd4cfcff14b02b707aa8303aece3a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:55.503892 15473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26f3c461ddfd4e30b2485ab0386c89b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:55.503892 13165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26f3c461ddfd4e30b2485ab0386c89b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:55.505997 24686 tablet_bootstrap.cc:492] T 26f3c461ddfd4e30b2485ab0386c89b2 P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:55.507268 24793 tablet_bootstrap.cc:492] T 96b5182e1e64427eae541e3c0a78b20a P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:55.507983 24793 tablet_bootstrap.cc:654] T 96b5182e1e64427eae541e3c0a78b20a P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:55.509043 24793 tablet_bootstrap.cc:492] T 96b5182e1e64427eae541e3c0a78b20a P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:55.509121 24793 ts_tablet_manager.cc:1367] T 96b5182e1e64427eae541e3c0a78b20a P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:55.509536 24793 raft_consensus.cc:348] T 96b5182e1e64427eae541e3c0a78b20a P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:55.509636 24793 raft_consensus.cc:374] T 96b5182e1e64427eae541e3c0a78b20a P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:55.509662 24793 raft_consensus.cc:729] T 96b5182e1e64427eae541e3c0a78b20a P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:55.511236 24793 consensus_queue.cc:260] T 96b5182e1e64427eae541e3c0a78b20a P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:55.511988 24793 ts_tablet_manager.cc:1398] T 96b5182e1e64427eae541e3c0a78b20a P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:56:55.514918 13051 raft_consensus.cc:1261] T 3fd497110cdb46fa9b51dd18e5408a5e P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:55.517717 24059 consensus_queue.cc:1035] T 3fd497110cdb46fa9b51dd18e5408a5e P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:55.506410 13163 tablet_service.cc:1467] Processing CreateTablet for tablet 96b5182e1e64427eae541e3c0a78b20a (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=7b7bd4cfcff14b02b707aa8303aece3a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:55.518380 13163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96b5182e1e64427eae541e3c0a78b20a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:55.506956 24686 tablet_bootstrap.cc:654] T 26f3c461ddfd4e30b2485ab0386c89b2 P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:55.520717 24796 tablet_bootstrap.cc:492] T 26f3c461ddfd4e30b2485ab0386c89b2 P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:55.512893 13184 raft_consensus.cc:1261] T 3fd497110cdb46fa9b51dd18e5408a5e P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:55.521930 24686 tablet_bootstrap.cc:492] T 26f3c461ddfd4e30b2485ab0386c89b2 P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:55.522024 24686 ts_tablet_manager.cc:1367] T 26f3c461ddfd4e30b2485ab0386c89b2 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.002s
I20240503 13:56:55.522065 24796 tablet_bootstrap.cc:654] T 26f3c461ddfd4e30b2485ab0386c89b2 P 61b887f0d40d497badeca5604c1f0546: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:55.522481 24686 raft_consensus.cc:348] T 26f3c461ddfd4e30b2485ab0386c89b2 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:55.522609 24686 raft_consensus.cc:374] T 26f3c461ddfd4e30b2485ab0386c89b2 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:55.522643 24686 raft_consensus.cc:729] T 26f3c461ddfd4e30b2485ab0386c89b2 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:55.522745 24686 consensus_queue.cc:260] T 26f3c461ddfd4e30b2485ab0386c89b2 P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:55.523021 24686 ts_tablet_manager.cc:1398] T 26f3c461ddfd4e30b2485ab0386c89b2 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:55.523187 24686 tablet_bootstrap.cc:492] T 96b5182e1e64427eae541e3c0a78b20a P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:55.523169 24796 tablet_bootstrap.cc:492] T 26f3c461ddfd4e30b2485ab0386c89b2 P 61b887f0d40d497badeca5604c1f0546: No bootstrap required, opened a new log
I20240503 13:56:55.523777 24796 ts_tablet_manager.cc:1367] T 26f3c461ddfd4e30b2485ab0386c89b2 P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:56:55.524086 24686 tablet_bootstrap.cc:654] T 96b5182e1e64427eae541e3c0a78b20a P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:55.524645 24796 raft_consensus.cc:348] T 26f3c461ddfd4e30b2485ab0386c89b2 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:55.524749 24796 raft_consensus.cc:374] T 26f3c461ddfd4e30b2485ab0386c89b2 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:55.524781 24796 raft_consensus.cc:729] T 26f3c461ddfd4e30b2485ab0386c89b2 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:55.524916 24796 consensus_queue.cc:260] T 26f3c461ddfd4e30b2485ab0386c89b2 P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:55.525269 24796 ts_tablet_manager.cc:1398] T 26f3c461ddfd4e30b2485ab0386c89b2 P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:55.525439 12905 catalog_manager.cc:5576] TS 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695) does not have the latest schema for tablet 3fd497110cdb46fa9b51dd18e5408a5e (table rebalancer_dynamic_table [id=089c57df50994be18a8170195960a2f5]). Expected version 1 got 0
I20240503 13:56:55.525751 24687 tablet_bootstrap.cc:492] T 5140b9d0611646fda51207df60c8a4df P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:55.525843 24687 ts_tablet_manager.cc:1367] T 5140b9d0611646fda51207df60c8a4df P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.000s
I20240503 13:56:55.513890 13164 tablet_service.cc:1467] Processing CreateTablet for tablet 5140b9d0611646fda51207df60c8a4df (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=7b7bd4cfcff14b02b707aa8303aece3a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:55.526268 13164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5140b9d0611646fda51207df60c8a4df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:55.526312 24687 raft_consensus.cc:348] T 5140b9d0611646fda51207df60c8a4df P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:55.526419 24687 raft_consensus.cc:374] T 5140b9d0611646fda51207df60c8a4df P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:55.526451 24687 raft_consensus.cc:729] T 5140b9d0611646fda51207df60c8a4df P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:55.526567 24687 consensus_queue.cc:260] T 5140b9d0611646fda51207df60c8a4df P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:55.526834 24687 ts_tablet_manager.cc:1398] T 5140b9d0611646fda51207df60c8a4df P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:55.526988 24687 tablet_bootstrap.cc:492] T 96b5182e1e64427eae541e3c0a78b20a P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:55.527855 24687 tablet_bootstrap.cc:654] T 96b5182e1e64427eae541e3c0a78b20a P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:55.528541 24059 consensus_queue.cc:1035] T 3fd497110cdb46fa9b51dd18e5408a5e P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:55.533910 24804 tablet_bootstrap.cc:492] T 5140b9d0611646fda51207df60c8a4df P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:55.534816 24687 tablet_bootstrap.cc:492] T 96b5182e1e64427eae541e3c0a78b20a P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:55.534888 24687 ts_tablet_manager.cc:1367] T 96b5182e1e64427eae541e3c0a78b20a P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 13:56:55.535383 24687 raft_consensus.cc:348] T 96b5182e1e64427eae541e3c0a78b20a P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:55.535514 24687 raft_consensus.cc:374] T 96b5182e1e64427eae541e3c0a78b20a P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:55.535547 24687 raft_consensus.cc:729] T 96b5182e1e64427eae541e3c0a78b20a P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:55.535661 24687 consensus_queue.cc:260] T 96b5182e1e64427eae541e3c0a78b20a P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:55.535909 24687 ts_tablet_manager.cc:1398] T 96b5182e1e64427eae541e3c0a78b20a P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:55.537367 24804 tablet_bootstrap.cc:654] T 5140b9d0611646fda51207df60c8a4df P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:55.538441 24804 tablet_bootstrap.cc:492] T 5140b9d0611646fda51207df60c8a4df P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:55.538522 24804 ts_tablet_manager.cc:1367] T 5140b9d0611646fda51207df60c8a4df P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:56:55.538928 24804 raft_consensus.cc:348] T 5140b9d0611646fda51207df60c8a4df P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:55.539038 24804 raft_consensus.cc:374] T 5140b9d0611646fda51207df60c8a4df P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:55.539065 24804 raft_consensus.cc:729] T 5140b9d0611646fda51207df60c8a4df P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:55.539180 24804 consensus_queue.cc:260] T 5140b9d0611646fda51207df60c8a4df P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:55.539464 24804 ts_tablet_manager.cc:1398] T 5140b9d0611646fda51207df60c8a4df P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:55.541123 24795 tablet_bootstrap.cc:492] T 26f3c461ddfd4e30b2485ab0386c89b2 P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:55.542093 24795 tablet_bootstrap.cc:654] T 26f3c461ddfd4e30b2485ab0386c89b2 P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:55.543320 12910 catalog_manager.cc:5576] TS 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695) does not have the latest schema for tablet 3fd497110cdb46fa9b51dd18e5408a5e (table rebalancer_dynamic_table [id=089c57df50994be18a8170195960a2f5]). Expected version 1 got 0
I20240503 13:56:55.544445 12907 catalog_manager.cc:5576] TS 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387) does not have the latest schema for tablet 3fd497110cdb46fa9b51dd18e5408a5e (table rebalancer_dynamic_table [id=089c57df50994be18a8170195960a2f5]). Expected version 1 got 0
I20240503 13:56:55.547173 12903 catalog_manager.cc:5576] TS 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387) does not have the latest schema for tablet 3fd497110cdb46fa9b51dd18e5408a5e (table rebalancer_dynamic_table [id=089c57df50994be18a8170195960a2f5]). Expected version 1 got 0
I20240503 13:56:55.548620 24795 tablet_bootstrap.cc:492] T 26f3c461ddfd4e30b2485ab0386c89b2 P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:55.548767 24795 ts_tablet_manager.cc:1367] T 26f3c461ddfd4e30b2485ab0386c89b2 P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.001s
I20240503 13:56:55.549458 24795 raft_consensus.cc:348] T 26f3c461ddfd4e30b2485ab0386c89b2 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:55.549643 24795 raft_consensus.cc:374] T 26f3c461ddfd4e30b2485ab0386c89b2 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:55.549714 24795 raft_consensus.cc:729] T 26f3c461ddfd4e30b2485ab0386c89b2 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:55.549888 24795 consensus_queue.cc:260] T 26f3c461ddfd4e30b2485ab0386c89b2 P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:55.550395 24803 tablet.cc:1722] T 3fd497110cdb46fa9b51dd18e5408a5e P ba0194098aa84449ae17e8a0d4c43703: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:55.551505 24803 tablet.cc:1620] T 3fd497110cdb46fa9b51dd18e5408a5e P ba0194098aa84449ae17e8a0d4c43703: MemRowSet was empty: no flush needed.
I20240503 13:56:55.552865 24686 tablet_bootstrap.cc:492] T 96b5182e1e64427eae541e3c0a78b20a P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:55.552966 24686 ts_tablet_manager.cc:1367] T 96b5182e1e64427eae541e3c0a78b20a P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.030s user 0.000s sys 0.001s
I20240503 13:56:55.553088 24795 ts_tablet_manager.cc:1398] T 26f3c461ddfd4e30b2485ab0386c89b2 P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240503 13:56:55.553450 24686 raft_consensus.cc:348] T 96b5182e1e64427eae541e3c0a78b20a P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:55.553573 24686 raft_consensus.cc:374] T 96b5182e1e64427eae541e3c0a78b20a P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:55.553601 24686 raft_consensus.cc:729] T 96b5182e1e64427eae541e3c0a78b20a P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:55.553706 24686 consensus_queue.cc:260] T 96b5182e1e64427eae541e3c0a78b20a P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:55.553965 24686 ts_tablet_manager.cc:1398] T 96b5182e1e64427eae541e3c0a78b20a P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:55.554128 24686 tablet_bootstrap.cc:492] T 5140b9d0611646fda51207df60c8a4df P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:55.554270 24811 tablet.cc:1722] T 3fd497110cdb46fa9b51dd18e5408a5e P 8ccf5a4ac450414c9d3579bd5800b770: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:55.554621 24811 tablet.cc:1620] T 3fd497110cdb46fa9b51dd18e5408a5e P 8ccf5a4ac450414c9d3579bd5800b770: MemRowSet was empty: no flush needed.
I20240503 13:56:55.555371 24686 tablet_bootstrap.cc:654] T 5140b9d0611646fda51207df60c8a4df P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:55.547691 24808 tablet.cc:1722] T 3fd497110cdb46fa9b51dd18e5408a5e P 60ba3ffe3b7746469834cd91c84be14e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:55.563920 24808 tablet.cc:1620] T 3fd497110cdb46fa9b51dd18e5408a5e P 60ba3ffe3b7746469834cd91c84be14e: MemRowSet was empty: no flush needed.
I20240503 13:56:55.564894 12905 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=089c57df50994be18a8170195960a2f5] alter complete (version 1)
I20240503 13:56:55.577533 24686 tablet_bootstrap.cc:492] T 5140b9d0611646fda51207df60c8a4df P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:55.577678 24686 ts_tablet_manager.cc:1367] T 5140b9d0611646fda51207df60c8a4df P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.000s
I20240503 13:56:55.578166 24686 raft_consensus.cc:348] T 5140b9d0611646fda51207df60c8a4df P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:55.578280 24686 raft_consensus.cc:374] T 5140b9d0611646fda51207df60c8a4df P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:55.578312 24686 raft_consensus.cc:729] T 5140b9d0611646fda51207df60c8a4df P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:55.578429 24686 consensus_queue.cc:260] T 5140b9d0611646fda51207df60c8a4df P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:55.578778 24686 ts_tablet_manager.cc:1398] T 5140b9d0611646fda51207df60c8a4df P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:55.597237 13184 consensus_queue.cc:237] T df28831df92d4c12b63b2f7da6c6449a P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:55.598568 15758 raft_consensus.cc:1261] T df28831df92d4c12b63b2f7da6c6449a P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Refusing update from remote peer 8ccf5a4ac450414c9d3579bd5800b770: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:56:55.598659 13051 raft_consensus.cc:1261] T df28831df92d4c12b63b2f7da6c6449a P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Refusing update from remote peer 8ccf5a4ac450414c9d3579bd5800b770: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:56:55.599002 24235 consensus_queue.cc:1035] T df28831df92d4c12b63b2f7da6c6449a P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:55.599190 24518 consensus_queue.cc:1035] T df28831df92d4c12b63b2f7da6c6449a P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:55.600283 23686 raft_consensus.cc:482] T 96b5182e1e64427eae541e3c0a78b20a P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:55.600371 23686 raft_consensus.cc:504] T 96b5182e1e64427eae541e3c0a78b20a P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:55.600728 23686 leader_election.cc:288] T 96b5182e1e64427eae541e3c0a78b20a P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387), ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:55.601253 24518 raft_consensus.cc:2932] T df28831df92d4c12b63b2f7da6c6449a P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:55.601366 13315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96b5182e1e64427eae541e3c0a78b20a" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703" is_pre_election: true
I20240503 13:56:55.601519 13315 raft_consensus.cc:2454] T 96b5182e1e64427eae541e3c0a78b20a P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 0.
I20240503 13:56:55.601820 15694 leader_election.cc:302] T 96b5182e1e64427eae541e3c0a78b20a P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76c386bef61a4bd5a907266a5118ffe2, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:55.601979 23686 raft_consensus.cc:2781] T 96b5182e1e64427eae541e3c0a78b20a P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:55.602025 23686 raft_consensus.cc:482] T 96b5182e1e64427eae541e3c0a78b20a P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:55.602052 23686 raft_consensus.cc:3037] T 96b5182e1e64427eae541e3c0a78b20a P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:55.602923 23686 raft_consensus.cc:504] T 96b5182e1e64427eae541e3c0a78b20a P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:55.603248 23686 leader_election.cc:288] T 96b5182e1e64427eae541e3c0a78b20a P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 election: Requested vote from peers 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387), ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:55.603499 15758 raft_consensus.cc:2932] T df28831df92d4c12b63b2f7da6c6449a P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:55.603595 13315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96b5182e1e64427eae541e3c0a78b20a" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703"
I20240503 13:56:55.603703 13315 raft_consensus.cc:3037] T 96b5182e1e64427eae541e3c0a78b20a P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:55.604672 13315 raft_consensus.cc:2454] T 96b5182e1e64427eae541e3c0a78b20a P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 1.
I20240503 13:56:55.605474 13184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96b5182e1e64427eae541e3c0a78b20a" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770" is_pre_election: true
I20240503 13:56:55.605533 15694 leader_election.cc:302] T 96b5182e1e64427eae541e3c0a78b20a P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76c386bef61a4bd5a907266a5118ffe2, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:55.605592 13184 raft_consensus.cc:2454] T 96b5182e1e64427eae541e3c0a78b20a P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 0.
I20240503 13:56:55.605736 23686 raft_consensus.cc:2781] T 96b5182e1e64427eae541e3c0a78b20a P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:55.605810 23686 raft_consensus.cc:686] T 96b5182e1e64427eae541e3c0a78b20a P 76c386bef61a4bd5a907266a5118ffe2 [term 1 LEADER]: Becoming Leader. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Running, Role: LEADER
I20240503 13:56:55.605865 13185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96b5182e1e64427eae541e3c0a78b20a" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770"
I20240503 13:56:55.605934 23686 consensus_queue.cc:237] T 96b5182e1e64427eae541e3c0a78b20a P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:55.605973 13185 raft_consensus.cc:3037] T 96b5182e1e64427eae541e3c0a78b20a P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:55.606554 12905 catalog_manager.cc:5484] T df28831df92d4c12b63b2f7da6c6449a P 8ccf5a4ac450414c9d3579bd5800b770 reported cstate change: config changed from index 4 to 5, VOTER ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) evicted. New cstate: current_term: 1 leader_uuid: "8ccf5a4ac450414c9d3579bd5800b770" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:55.610558 13051 raft_consensus.cc:2932] T df28831df92d4c12b63b2f7da6c6449a P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:55.612197 13295 tablet_service.cc:1514] Processing DeleteTablet for tablet df28831df92d4c12b63b2f7da6c6449a with delete_type TABLET_DATA_TOMBSTONED (TS ba0194098aa84449ae17e8a0d4c43703 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:33250
I20240503 13:56:55.612421 24523 tablet_replica.cc:330] T df28831df92d4c12b63b2f7da6c6449a P ba0194098aa84449ae17e8a0d4c43703: stopping tablet replica
I20240503 13:56:55.612634 24523 raft_consensus.cc:2229] T df28831df92d4c12b63b2f7da6c6449a P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:55.612787 24523 raft_consensus.cc:2258] T df28831df92d4c12b63b2f7da6c6449a P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:55.613215 24523 ts_tablet_manager.cc:1875] T df28831df92d4c12b63b2f7da6c6449a P ba0194098aa84449ae17e8a0d4c43703: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:56:55.615342 24523 ts_tablet_manager.cc:1888] T df28831df92d4c12b63b2f7da6c6449a P ba0194098aa84449ae17e8a0d4c43703: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 13:56:55.615407 24523 log.cc:1196] T df28831df92d4c12b63b2f7da6c6449a P ba0194098aa84449ae17e8a0d4c43703: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/wal/wals/df28831df92d4c12b63b2f7da6c6449a
I20240503 13:56:55.615913 12899 catalog_manager.cc:4830] TS ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279): tablet df28831df92d4c12b63b2f7da6c6449a (table rebalance_test_table_1 [id=ba90ff8253204f25829d57a56c0b5e94]) successfully deleted
I20240503 13:56:55.616860 13185 raft_consensus.cc:2454] T 96b5182e1e64427eae541e3c0a78b20a P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 1.
I20240503 13:56:55.621219 12909 catalog_manager.cc:5484] T 96b5182e1e64427eae541e3c0a78b20a P 76c386bef61a4bd5a907266a5118ffe2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71). New cstate: current_term: 1 leader_uuid: "76c386bef61a4bd5a907266a5118ffe2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:55.644459 13446 consensus_queue.cc:237] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:55.646225 13051 raft_consensus.cc:1261] T c047c61f42f04484842125847ca677aa P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Refusing update from remote peer 7690ee79577c45e3a2c8f8b552c0bff8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:56:55.646608 24855 consensus_queue.cc:1035] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 13:56:55.647022 13380 consensus_peers.cc:485] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c047c61f42f04484842125847ca677aa. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:55.647076 13316 raft_consensus.cc:1261] T c047c61f42f04484842125847ca677aa P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Refusing update from remote peer 7690ee79577c45e3a2c8f8b552c0bff8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:56:55.647953 24855 consensus_queue.cc:1035] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:55.650055 23971 raft_consensus.cc:2932] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:55.651831 13316 raft_consensus.cc:2932] T c047c61f42f04484842125847ca677aa P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:55.653367 12903 catalog_manager.cc:5484] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 reported cstate change: config changed from index -1 to 4, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) added. New cstate: current_term: 1 leader_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:55.653810 13051 raft_consensus.cc:2932] T c047c61f42f04484842125847ca677aa P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:55.673215 12909 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:56:55.679829 24803 tablet.cc:1722] T 3fd497110cdb46fa9b51dd18e5408a5e P ba0194098aa84449ae17e8a0d4c43703: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:55.680182 24803 tablet.cc:1620] T 3fd497110cdb46fa9b51dd18e5408a5e P ba0194098aa84449ae17e8a0d4c43703: MemRowSet was empty: no flush needed.
I20240503 13:56:55.680420 24811 tablet.cc:1722] T 3fd497110cdb46fa9b51dd18e5408a5e P 8ccf5a4ac450414c9d3579bd5800b770: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:55.680757 24811 tablet.cc:1620] T 3fd497110cdb46fa9b51dd18e5408a5e P 8ccf5a4ac450414c9d3579bd5800b770: MemRowSet was empty: no flush needed.
I20240503 13:56:55.681180 24808 tablet.cc:1722] T 3fd497110cdb46fa9b51dd18e5408a5e P 60ba3ffe3b7746469834cd91c84be14e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:55.681494 24808 tablet.cc:1620] T 3fd497110cdb46fa9b51dd18e5408a5e P 60ba3ffe3b7746469834cd91c84be14e: MemRowSet was empty: no flush needed.
I20240503 13:56:55.685846 12903 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=089c57df50994be18a8170195960a2f5] alter complete (version 2)
I20240503 13:56:55.733198 24077 raft_consensus.cc:482] T 26f3c461ddfd4e30b2485ab0386c89b2 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:55.733335 24077 raft_consensus.cc:504] T 26f3c461ddfd4e30b2485ab0386c89b2 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:55.733837 24077 leader_election.cc:288] T 26f3c461ddfd4e30b2485ab0386c89b2 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387), 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153)
I20240503 13:56:55.734429 13182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26f3c461ddfd4e30b2485ab0386c89b2" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770" is_pre_election: true
I20240503 13:56:55.734568 13182 raft_consensus.cc:2454] T 26f3c461ddfd4e30b2485ab0386c89b2 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 0.
I20240503 13:56:55.734638 24059 raft_consensus.cc:482] T 5140b9d0611646fda51207df60c8a4df P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:55.734723 24059 raft_consensus.cc:504] T 5140b9d0611646fda51207df60c8a4df P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:55.735109 24059 leader_election.cc:288] T 5140b9d0611646fda51207df60c8a4df P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:55.735714 15561 leader_election.cc:302] T 26f3c461ddfd4e30b2485ab0386c89b2 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 8ccf5a4ac450414c9d3579bd5800b770; no voters:
I20240503 13:56:55.736006 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5140b9d0611646fda51207df60c8a4df" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
I20240503 13:56:55.736059 15493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26f3c461ddfd4e30b2485ab0386c89b2" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546" is_pre_election: true
I20240503 13:56:55.736135 13446 raft_consensus.cc:2454] T 5140b9d0611646fda51207df60c8a4df P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 0.
I20240503 13:56:55.736166 15493 raft_consensus.cc:2454] T 26f3c461ddfd4e30b2485ab0386c89b2 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 0.
I20240503 13:56:55.737084 24077 raft_consensus.cc:2781] T 26f3c461ddfd4e30b2485ab0386c89b2 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:55.737145 24077 raft_consensus.cc:482] T 26f3c461ddfd4e30b2485ab0386c89b2 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:55.737181 24077 raft_consensus.cc:3037] T 26f3c461ddfd4e30b2485ab0386c89b2 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:55.737413 13249 leader_election.cc:302] T 5140b9d0611646fda51207df60c8a4df P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:55.737744 23473 raft_consensus.cc:2781] T 5140b9d0611646fda51207df60c8a4df P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:55.737790 23473 raft_consensus.cc:482] T 5140b9d0611646fda51207df60c8a4df P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:55.737821 23473 raft_consensus.cc:3037] T 5140b9d0611646fda51207df60c8a4df P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:55.738286 24077 raft_consensus.cc:504] T 26f3c461ddfd4e30b2485ab0386c89b2 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:55.738605 24077 leader_election.cc:288] T 26f3c461ddfd4e30b2485ab0386c89b2 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 election: Requested vote from peers 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387), 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153)
I20240503 13:56:55.738816 23473 raft_consensus.cc:504] T 5140b9d0611646fda51207df60c8a4df P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:55.739145 23473 leader_election.cc:288] T 5140b9d0611646fda51207df60c8a4df P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: Requested vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:55.739290 15493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26f3c461ddfd4e30b2485ab0386c89b2" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546"
I20240503 13:56:55.739396 15493 raft_consensus.cc:3037] T 26f3c461ddfd4e30b2485ab0386c89b2 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:55.739970 13185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5140b9d0611646fda51207df60c8a4df" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770" is_pre_election: true
I20240503 13:56:55.740114 13185 raft_consensus.cc:2454] T 5140b9d0611646fda51207df60c8a4df P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 0.
I20240503 13:56:55.740286 13183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5140b9d0611646fda51207df60c8a4df" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770"
I20240503 13:56:55.740379 15493 raft_consensus.cc:2454] T 26f3c461ddfd4e30b2485ab0386c89b2 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 1.
I20240503 13:56:55.740412 13183 raft_consensus.cc:3037] T 5140b9d0611646fda51207df60c8a4df P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:55.741426 13183 raft_consensus.cc:2454] T 5140b9d0611646fda51207df60c8a4df P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 1.
I20240503 13:56:55.742813 15561 leader_election.cc:302] T 26f3c461ddfd4e30b2485ab0386c89b2 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 61b887f0d40d497badeca5604c1f0546; no voters:
I20240503 13:56:55.743022 24077 raft_consensus.cc:2781] T 26f3c461ddfd4e30b2485ab0386c89b2 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:55.743098 24077 raft_consensus.cc:686] T 26f3c461ddfd4e30b2485ab0386c89b2 P 009f849ee7b54c80a3f5264695047e4d [term 1 LEADER]: Becoming Leader. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Running, Role: LEADER
I20240503 13:56:55.743228 24077 consensus_queue.cc:237] T 26f3c461ddfd4e30b2485ab0386c89b2 P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:55.743456 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5140b9d0611646fda51207df60c8a4df" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
I20240503 13:56:55.743587 13446 raft_consensus.cc:3037] T 5140b9d0611646fda51207df60c8a4df P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:55.744597 13446 raft_consensus.cc:2454] T 5140b9d0611646fda51207df60c8a4df P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 1.
I20240503 13:56:55.744894 13182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26f3c461ddfd4e30b2485ab0386c89b2" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770"
I20240503 13:56:55.745009 13182 raft_consensus.cc:3037] T 26f3c461ddfd4e30b2485ab0386c89b2 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:55.745993 13182 raft_consensus.cc:2454] T 26f3c461ddfd4e30b2485ab0386c89b2 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 1.
I20240503 13:56:55.747788 13249 leader_election.cc:302] T 5140b9d0611646fda51207df60c8a4df P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:55.747996 23473 raft_consensus.cc:2781] T 5140b9d0611646fda51207df60c8a4df P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:55.748097 23473 raft_consensus.cc:686] T 5140b9d0611646fda51207df60c8a4df P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Becoming Leader. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Running, Role: LEADER
I20240503 13:56:55.748231 23473 consensus_queue.cc:237] T 5140b9d0611646fda51207df60c8a4df P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:55.748477 12906 catalog_manager.cc:5484] T 26f3c461ddfd4e30b2485ab0386c89b2 P 009f849ee7b54c80a3f5264695047e4d reported cstate change: term changed from 0 to 1, leader changed from <none> to 009f849ee7b54c80a3f5264695047e4d (127.6.98.70). New cstate: current_term: 1 leader_uuid: "009f849ee7b54c80a3f5264695047e4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:55.751883 12912 catalog_manager.cc:5484] T 5140b9d0611646fda51207df60c8a4df P ba0194098aa84449ae17e8a0d4c43703 reported cstate change: term changed from 0 to 1, leader changed from <none> to ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67). New cstate: current_term: 1 leader_uuid: "ba0194098aa84449ae17e8a0d4c43703" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } }
I20240503 13:56:55.753340 15889 raft_consensus.cc:1203] T 7b57f64985b74e5fa75a5603ab8cc4bf P e47105a0f46e4511b7112e571688a4c7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:56:55.788064 12906 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:56:55.794948 24803 tablet.cc:1722] T 3fd497110cdb46fa9b51dd18e5408a5e P ba0194098aa84449ae17e8a0d4c43703: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:55.795297 24811 tablet.cc:1722] T 3fd497110cdb46fa9b51dd18e5408a5e P 8ccf5a4ac450414c9d3579bd5800b770: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:55.801839 24808 tablet.cc:1722] T 3fd497110cdb46fa9b51dd18e5408a5e P 60ba3ffe3b7746469834cd91c84be14e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:55.807535 12906 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table_new_name [id=089c57df50994be18a8170195960a2f5] alter complete (version 3)
I20240503 13:56:55.810323 13163 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 13:56:55.814931 13034 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:56:55.821645 15604 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 13:56:55.828563 15738 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 13:56:55.829484 13424 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 13:56:55.842224 15473 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:56:55.842535 13295 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 13:56:55.869299 15869 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 13:56:55.915102 12910 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:55.915275 12910 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:55.917611 12910 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Sending DeleteTablet for 3 replicas of tablet 3fd497110cdb46fa9b51dd18e5408a5e
I20240503 13:56:55.921376 13034 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fd497110cdb46fa9b51dd18e5408a5e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:55 UTC) from {username='slave'} at 127.0.0.1:59458
I20240503 13:56:55.922029 13163 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fd497110cdb46fa9b51dd18e5408a5e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:55 UTC) from {username='slave'} at 127.0.0.1:41668
I20240503 13:56:55.922444 13295 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fd497110cdb46fa9b51dd18e5408a5e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:55 UTC) from {username='slave'} at 127.0.0.1:33250
I20240503 13:56:55.922626 24523 tablet_replica.cc:330] T 3fd497110cdb46fa9b51dd18e5408a5e P ba0194098aa84449ae17e8a0d4c43703: stopping tablet replica
I20240503 13:56:55.922763 24523 raft_consensus.cc:2229] T 3fd497110cdb46fa9b51dd18e5408a5e P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:56:55.922963 24523 raft_consensus.cc:2258] T 3fd497110cdb46fa9b51dd18e5408a5e P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:55.923470 24926 tablet_replica.cc:330] T 3fd497110cdb46fa9b51dd18e5408a5e P 8ccf5a4ac450414c9d3579bd5800b770: stopping tablet replica
I20240503 13:56:55.923655 24926 raft_consensus.cc:2229] T 3fd497110cdb46fa9b51dd18e5408a5e P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:55.923719 24523 ts_tablet_manager.cc:1875] T 3fd497110cdb46fa9b51dd18e5408a5e P ba0194098aa84449ae17e8a0d4c43703: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:55.923844 24926 raft_consensus.cc:2258] T 3fd497110cdb46fa9b51dd18e5408a5e P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:55.924468 24926 ts_tablet_manager.cc:1875] T 3fd497110cdb46fa9b51dd18e5408a5e P 8ccf5a4ac450414c9d3579bd5800b770: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:55.925848 24523 ts_tablet_manager.cc:1888] T 3fd497110cdb46fa9b51dd18e5408a5e P ba0194098aa84449ae17e8a0d4c43703: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:55.925982 24523 log.cc:1196] T 3fd497110cdb46fa9b51dd18e5408a5e P ba0194098aa84449ae17e8a0d4c43703: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/wal/wals/3fd497110cdb46fa9b51dd18e5408a5e
I20240503 13:56:55.926716 24523 ts_tablet_manager.cc:1909] T 3fd497110cdb46fa9b51dd18e5408a5e P ba0194098aa84449ae17e8a0d4c43703: Deleting consensus metadata
I20240503 13:56:55.927459 12899 catalog_manager.cc:4830] TS ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279): tablet 3fd497110cdb46fa9b51dd18e5408a5e (table rebalancer_dynamic_table_new_name [id=089c57df50994be18a8170195960a2f5]) successfully deleted
I20240503 13:56:55.928320 24926 ts_tablet_manager.cc:1888] T 3fd497110cdb46fa9b51dd18e5408a5e P 8ccf5a4ac450414c9d3579bd5800b770: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:55.928444 24926 log.cc:1196] T 3fd497110cdb46fa9b51dd18e5408a5e P 8ccf5a4ac450414c9d3579bd5800b770: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-1/wal/wals/3fd497110cdb46fa9b51dd18e5408a5e
I20240503 13:56:55.929374 24926 ts_tablet_manager.cc:1909] T 3fd497110cdb46fa9b51dd18e5408a5e P 8ccf5a4ac450414c9d3579bd5800b770: Deleting consensus metadata
I20240503 13:56:55.930033 12899 catalog_manager.cc:4830] TS 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387): tablet 3fd497110cdb46fa9b51dd18e5408a5e (table rebalancer_dynamic_table_new_name [id=089c57df50994be18a8170195960a2f5]) successfully deleted
I20240503 13:56:55.944664 24925 tablet_replica.cc:330] T 3fd497110cdb46fa9b51dd18e5408a5e P 60ba3ffe3b7746469834cd91c84be14e: stopping tablet replica
I20240503 13:56:55.944833 24925 raft_consensus.cc:2229] T 3fd497110cdb46fa9b51dd18e5408a5e P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:55.945047 24925 raft_consensus.cc:2258] T 3fd497110cdb46fa9b51dd18e5408a5e P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:55.945833 24925 ts_tablet_manager.cc:1875] T 3fd497110cdb46fa9b51dd18e5408a5e P 60ba3ffe3b7746469834cd91c84be14e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:55.948299 24925 ts_tablet_manager.cc:1888] T 3fd497110cdb46fa9b51dd18e5408a5e P 60ba3ffe3b7746469834cd91c84be14e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:55.948367 24925 log.cc:1196] T 3fd497110cdb46fa9b51dd18e5408a5e P 60ba3ffe3b7746469834cd91c84be14e: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-0/wal/wals/3fd497110cdb46fa9b51dd18e5408a5e
I20240503 13:56:55.948681 24925 ts_tablet_manager.cc:1909] T 3fd497110cdb46fa9b51dd18e5408a5e P 60ba3ffe3b7746469834cd91c84be14e: Deleting consensus metadata
I20240503 13:56:55.949427 12897 catalog_manager.cc:4830] TS 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695): tablet 3fd497110cdb46fa9b51dd18e5408a5e (table rebalancer_dynamic_table_new_name [id=089c57df50994be18a8170195960a2f5]) successfully deleted
I20240503 13:56:56.019205 12905 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:56:56.025624 15604 tablet_service.cc:1467] Processing CreateTablet for tablet a8fc5c3dd305409faba2b9f1dcea0c65 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d2265169a02749789d7df005bb024892]), partition=
I20240503 13:56:56.025774 15473 tablet_service.cc:1467] Processing CreateTablet for tablet a8fc5c3dd305409faba2b9f1dcea0c65 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d2265169a02749789d7df005bb024892]), partition=
I20240503 13:56:56.025751 15738 tablet_service.cc:1467] Processing CreateTablet for tablet a8fc5c3dd305409faba2b9f1dcea0c65 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d2265169a02749789d7df005bb024892]), partition=
I20240503 13:56:56.025954 15604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8fc5c3dd305409faba2b9f1dcea0c65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:56.025978 15473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8fc5c3dd305409faba2b9f1dcea0c65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:56.026042 15738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8fc5c3dd305409faba2b9f1dcea0c65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:56.028117 24796 tablet_bootstrap.cc:492] T a8fc5c3dd305409faba2b9f1dcea0c65 P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:56.028131 24795 tablet_bootstrap.cc:492] T a8fc5c3dd305409faba2b9f1dcea0c65 P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:56.029084 24795 tablet_bootstrap.cc:654] T a8fc5c3dd305409faba2b9f1dcea0c65 P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:56.029103 24796 tablet_bootstrap.cc:654] T a8fc5c3dd305409faba2b9f1dcea0c65 P 61b887f0d40d497badeca5604c1f0546: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:56.032685 24796 tablet_bootstrap.cc:492] T a8fc5c3dd305409faba2b9f1dcea0c65 P 61b887f0d40d497badeca5604c1f0546: No bootstrap required, opened a new log
I20240503 13:56:56.032737 24795 tablet_bootstrap.cc:492] T a8fc5c3dd305409faba2b9f1dcea0c65 P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:56.032784 24796 ts_tablet_manager.cc:1367] T a8fc5c3dd305409faba2b9f1dcea0c65 P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240503 13:56:56.032833 24795 ts_tablet_manager.cc:1367] T a8fc5c3dd305409faba2b9f1dcea0c65 P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240503 13:56:56.033236 24795 raft_consensus.cc:348] T a8fc5c3dd305409faba2b9f1dcea0c65 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:56.033344 24795 raft_consensus.cc:374] T a8fc5c3dd305409faba2b9f1dcea0c65 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:56.033377 24795 raft_consensus.cc:729] T a8fc5c3dd305409faba2b9f1dcea0c65 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:56.033406 24796 raft_consensus.cc:348] T a8fc5c3dd305409faba2b9f1dcea0c65 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:56.033509 24796 raft_consensus.cc:374] T a8fc5c3dd305409faba2b9f1dcea0c65 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:56.033491 24795 consensus_queue.cc:260] T a8fc5c3dd305409faba2b9f1dcea0c65 P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:56.033540 24796 raft_consensus.cc:729] T a8fc5c3dd305409faba2b9f1dcea0c65 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:56.033650 24796 consensus_queue.cc:260] T a8fc5c3dd305409faba2b9f1dcea0c65 P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:56.033783 24795 ts_tablet_manager.cc:1398] T a8fc5c3dd305409faba2b9f1dcea0c65 P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:56.033948 24796 ts_tablet_manager.cc:1398] T a8fc5c3dd305409faba2b9f1dcea0c65 P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:56.035116 24957 tablet_bootstrap.cc:492] T a8fc5c3dd305409faba2b9f1dcea0c65 P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:56.036208 24957 tablet_bootstrap.cc:654] T a8fc5c3dd305409faba2b9f1dcea0c65 P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:56.037339 24957 tablet_bootstrap.cc:492] T a8fc5c3dd305409faba2b9f1dcea0c65 P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:56.037434 24957 ts_tablet_manager.cc:1367] T a8fc5c3dd305409faba2b9f1dcea0c65 P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 13:56:56.037892 24957 raft_consensus.cc:348] T a8fc5c3dd305409faba2b9f1dcea0c65 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:56.037999 24957 raft_consensus.cc:374] T a8fc5c3dd305409faba2b9f1dcea0c65 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:56.038031 24957 raft_consensus.cc:729] T a8fc5c3dd305409faba2b9f1dcea0c65 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:56.038157 24957 consensus_queue.cc:260] T a8fc5c3dd305409faba2b9f1dcea0c65 P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:56.038441 24957 ts_tablet_manager.cc:1398] T a8fc5c3dd305409faba2b9f1dcea0c65 P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:56.054419 15473 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:56:56.062417 13034 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:56:56.063087 15604 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 13:56:56.063840 13424 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 13:56:56.074312 13163 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 13:56:56.080353 15738 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 13:56:56.086982 24077 raft_consensus.cc:482] T a8fc5c3dd305409faba2b9f1dcea0c65 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:56.087102 24077 raft_consensus.cc:504] T a8fc5c3dd305409faba2b9f1dcea0c65 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:56.087440 24077 leader_election.cc:288] T a8fc5c3dd305409faba2b9f1dcea0c65 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153), 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885)
I20240503 13:56:56.088462 15492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fc5c3dd305409faba2b9f1dcea0c65" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546" is_pre_election: true
I20240503 13:56:56.088456 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fc5c3dd305409faba2b9f1dcea0c65" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2" is_pre_election: true
I20240503 13:56:56.088608 15492 raft_consensus.cc:2454] T a8fc5c3dd305409faba2b9f1dcea0c65 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 0.
I20240503 13:56:56.088619 15757 raft_consensus.cc:2454] T a8fc5c3dd305409faba2b9f1dcea0c65 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 0.
I20240503 13:56:56.089208 15561 leader_election.cc:302] T a8fc5c3dd305409faba2b9f1dcea0c65 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 61b887f0d40d497badeca5604c1f0546; no voters:
I20240503 13:56:56.089566 24901 raft_consensus.cc:2781] T a8fc5c3dd305409faba2b9f1dcea0c65 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:56.089624 24901 raft_consensus.cc:482] T a8fc5c3dd305409faba2b9f1dcea0c65 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:56.089658 24901 raft_consensus.cc:3037] T a8fc5c3dd305409faba2b9f1dcea0c65 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:56.090764 24901 raft_consensus.cc:504] T a8fc5c3dd305409faba2b9f1dcea0c65 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:56.091125 24901 leader_election.cc:288] T a8fc5c3dd305409faba2b9f1dcea0c65 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 election: Requested vote from peers 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153), 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885)
I20240503 13:56:56.091694 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fc5c3dd305409faba2b9f1dcea0c65" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2"
I20240503 13:56:56.091764 15492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fc5c3dd305409faba2b9f1dcea0c65" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546"
I20240503 13:56:56.091825 15757 raft_consensus.cc:3037] T a8fc5c3dd305409faba2b9f1dcea0c65 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:56.091864 15492 raft_consensus.cc:3037] T a8fc5c3dd305409faba2b9f1dcea0c65 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:56.092834 15492 raft_consensus.cc:2454] T a8fc5c3dd305409faba2b9f1dcea0c65 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 1.
I20240503 13:56:56.092938 15757 raft_consensus.cc:2454] T a8fc5c3dd305409faba2b9f1dcea0c65 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 1.
I20240503 13:56:56.093282 15561 leader_election.cc:302] T a8fc5c3dd305409faba2b9f1dcea0c65 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 61b887f0d40d497badeca5604c1f0546; no voters:
I20240503 13:56:56.093566 24901 raft_consensus.cc:2781] T a8fc5c3dd305409faba2b9f1dcea0c65 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:56.093636 24901 raft_consensus.cc:686] T a8fc5c3dd305409faba2b9f1dcea0c65 P 009f849ee7b54c80a3f5264695047e4d [term 1 LEADER]: Becoming Leader. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Running, Role: LEADER
I20240503 13:56:56.093745 24901 consensus_queue.cc:237] T a8fc5c3dd305409faba2b9f1dcea0c65 P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:56.094863 13295 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 13:56:56.089007 24973 ts_tablet_manager.cc:902] T c047c61f42f04484842125847ca677aa P e47105a0f46e4511b7112e571688a4c7: Initiating tablet copy from peer 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
F20240503 13:56:56.096694 24973 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-397f0e3e877d4acd9157bdf99bee6255.queue_time_us, tablet-download-pool-397f0e3e877d4acd9157bdf99bee6255.queue_time_us) (tablet-download-pool-397f0e3e877d4acd9157bdf99bee6255.run_cpu_time_us, tablet-download-pool-397f0e3e877d4acd9157bdf99bee6255.run_cpu_time_us) (tablet-download-pool-397f0e3e877d4acd9157bdf99bee6255.run_wall_time_us, tablet-download-pool-397f0e3e877d4acd9157bdf99bee6255.run_wall_time_us) (tablet-download-pool-477c33f203f649019eff2b28524fad90.queue_time_us, tablet-download-pool-477c33f203f649019eff2b28524fad90.queue_time_us) (tablet-download-pool-477c33f203f649019eff2b28524fad90.run_cpu_time_us, tablet-download-pool-477c33f203f649019eff2b28524fad90.run_cpu_time_us) (tablet-download-pool-477c33f203f649019eff2b28524fad90.run_wall_time_us, tablet-download-pool-477c33f203f649019eff2b28524fad90.run_wall_time_us) (tablet-download-pool-4bd42569fd3b4b39b0bd489acebbad10.queue_time_us, tablet-download-pool-4bd42569fd3b4b39b0bd489acebbad10.queue_time_us) (tablet-download-pool-4bd42569fd3b4b39b0bd489acebbad10.run_cpu_time_us, tablet-download-pool-4bd42569fd3b4b39b0bd489acebbad10.run_cpu_time_us) (tablet-download-pool-4bd42569fd3b4b39b0bd489acebbad10.run_wall_time_us, tablet-download-pool-4bd42569fd3b4b39b0bd489acebbad10.run_wall_time_us) (tablet-download-pool-604875eff3594876af54e9126057bd7f.queue_time_us, tablet-download-pool-604875eff3594876af54e9126057bd7f.queue_time_us) (tablet-download-pool-604875eff3594876af54e9126057bd7f.run_cpu_time_us, tablet-download-pool-604875eff3594876af54e9126057bd7f.run_cpu_time_us) (tablet-download-pool-604875eff3594876af54e9126057bd7f.run_wall_time_us, tablet-download-pool-604875eff3594876af54e9126057bd7f.run_wall_time_us) (tablet-download-pool-7a1c36d4fd0049139c98db7af595bc62.queue_time_us, tablet-download-pool-7a1c36d4fd0049139c98db7af595bc62.queue_time_us) (tablet-download-pool-7a1c36d4fd0049139c98db7af595bc62.run_cpu_time_us, tablet-download-pool-7a1c36d4fd0049139c98db7af595bc62.run_cpu_time_us) (tablet-download-pool-7a1c36d4fd0049139c98db7af595bc62.run_wall_time_us, tablet-download-pool-7a1c36d4fd0049139c98db7af595bc62.run_wall_time_us) (tablet-download-pool-7b57f64985b74e5fa75a5603ab8cc4bf.queue_time_us, tablet-download-pool-7b57f64985b74e5fa75a5603ab8cc4bf.queue_time_us) (tablet-download-pool-7b57f64985b74e5fa75a5603ab8cc4bf.run_cpu_time_us, tablet-download-pool-7b57f64985b74e5fa75a5603ab8cc4bf.run_cpu_time_us) (tablet-download-pool-7b57f64985b74e5fa75a5603ab8cc4bf.run_wall_time_us, tablet-download-pool-7b57f64985b74e5fa75a5603ab8cc4bf.run_wall_time_us) (tablet-download-pool-84f1ab11e74045f8a9d7033c5ead0c14.queue_time_us, tablet-download-pool-84f1ab11e74045f8a9d7033c5ead0c14.queue_time_us) (tablet-download-pool-84f1ab11e74045f8a9d7033c5ead0c14.run_cpu_time_us, tablet-download-pool-84f1ab11e74045f8a9d7033c5ead0c14.run_cpu_time_us) (tablet-download-pool-84f1ab11e74045f8a9d7033c5ead0c14.run_wall_time_us, tablet-download-pool-84f1ab11e74045f8a9d7033c5ead0c14.run_wall_time_us) (tablet-download-pool-866723d26350414197ec27544277db9c.queue_time_us, tablet-download-pool-866723d26350414197ec27544277db9c.queue_time_us) (tablet-download-pool-866723d26350414197ec27544277db9c.run_cpu_time_us, tablet-download-pool-866723d26350414197ec27544277db9c.run_cpu_time_us) (tablet-download-pool-866723d26350414197ec27544277db9c.run_wall_time_us, tablet-download-pool-866723d26350414197ec27544277db9c.run_wall_time_us) (tablet-download-pool-a746bcb2d718433f8c9acc8bd027a429.queue_time_us, tablet-download-pool-a746bcb2d718433f8c9acc8bd027a429.queue_time_us) (tablet-download-pool-a746bcb2d718433f8c9acc8bd027a429.run_cpu_time_us, tablet-download-pool-a746bcb2d718433f8c9acc8bd027a429.run_cpu_time_us) (tablet-download-pool-a746bcb2d718433f8c9acc8bd027a429.run_wall_time_us, tablet-download-pool-a746bcb2d718433f8c9acc8bd027a429.run_wall_time_us) (tablet-download-pool-b923fd3f226e409b813bb4e63325a7fc.queue_time_us, tablet-download-pool-b923fd3f226e409b813bb4e63325a7fc.queue_time_us) (tablet-download-pool-b923fd3f226e409b813bb4e63325a7fc.run_cpu_time_us, tablet-download-pool-b923fd3f226e409b813bb4e63325a7fc.run_cpu_time_us) (tablet-download-pool-b923fd3f226e409b813bb4e63325a7fc.run_wall_time_us, tablet-download-pool-b923fd3f226e409b813bb4e63325a7fc.run_wall_time_us) (tablet-download-pool-bb4f0bb47a29437090750c4486df2f02.queue_time_us, tablet-download-pool-bb4f0bb47a29437090750c4486df2f02.queue_time_us) (tablet-download-pool-bb4f0bb47a29437090750c4486df2f02.run_cpu_time_us, tablet-download-pool-bb4f0bb47a29437090750c4486df2f02.run_cpu_time_us) (tablet-download-pool-bb4f0bb47a29437090750c4486df2f02.run_wall_time_us, tablet-download-pool-bb4f0bb47a29437090750c4486df2f02.run_wall_time_us) (tablet-download-pool-c047c61f42f04484842125847ca677aa.queue_time_us, tablet-download-pool-c047c61f42f04484842125847ca677aa.queue_time_us) (tablet-download-pool-c7d5e8c09ec14ac7891c8a63cb896878.queue_time_us, tablet-download-pool-c7d5e8c09ec14ac7891c8a63cb896878.queue_time_us) (tablet-download-pool-c7d5e8c09ec14ac7891c8a63cb896878.run_cpu_time_us, tablet-download-pool-c7d5e8c09ec14ac7891c8a63cb896878.run_cpu_time_us) (tablet-download-pool-c7d5e8c09ec14ac7891c8a63cb896878.run_wall_time_us, tablet-download-pool-c7d5e8c09ec14ac7891c8a63cb896878.run_wall_time_us) (tablet-download-pool-cbd6d38ea18b484d80f4a918148efd3a.queue_time_us, tablet-download-pool-cbd6d38ea18b484d80f4a918148efd3a.queue_time_us) (tablet-download-pool-cbd6d38ea18b484d80f4a918148efd3a.run_cpu_time_us, tablet-download-pool-cbd6d38ea18b484d80f4a918148efd3a.run_cpu_time_us) (tablet-download-pool-cbd6d38ea18b484d80f4a918148efd3a.run_wall_time_us, tablet-download-pool-cbd6d38ea18b484d80f4a918148efd3a.run_wall_time_us) (tablet-download-pool-d50d05d432e84d5eb0e54d1a68cdf158.queue_time_us, tablet-download-pool-d50d05d432e84d5eb0e54d1a68cdf158.queue_time_us) (tablet-download-pool-d50d05d432e84d5eb0e54d1a68cdf158.run_cpu_time_us, tablet-download-pool-d50d05d432e84d5eb0e54d1a68cdf158.run_cpu_time_us) (tablet-download-pool-d50d05d432e84d5eb0e54d1a68cdf158.run_wall_time_us, tablet-download-pool-d50d05d432e84d5eb0e54d1a68cdf158.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
I20240503 13:56:56.096864 12906 catalog_manager.cc:5484] T a8fc5c3dd305409faba2b9f1dcea0c65 P 009f849ee7b54c80a3f5264695047e4d reported cstate change: term changed from 0 to 1, leader changed from <none> to 009f849ee7b54c80a3f5264695047e4d (127.6.98.70). New cstate: current_term: 1 leader_uuid: "009f849ee7b54c80a3f5264695047e4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:56.102263 15869 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
*** Check failure stack trace: ***
@ 0x7fc4bfbcbdcd google::LogMessage::Fail() at ??:0
@ 0x7fc4bfbcfb93 google::LogMessage::SendToLog() at ??:0
I20240503 13:56:56.119029 23686 consensus_queue.cc:1035] T 96b5182e1e64427eae541e3c0a78b20a P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
@ 0x7fc4bfbcb7cc google::LogMessage::Flush() at ??:0
I20240503 13:56:56.125284 24901 consensus_queue.cc:1035] T 26f3c461ddfd4e30b2485ab0386c89b2 P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
@ 0x7fc4bfbccf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fc4c1035044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7fc4c1019f98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7fc4c1017efc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7fc4c3bca77a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240503 13:56:56.149360 23621 consensus_queue.cc:1035] T 26f3c461ddfd4e30b2485ab0386c89b2 P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:56.154536 24959 consensus_queue.cc:1035] T 96b5182e1e64427eae541e3c0a78b20a P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
@ 0x7fc4c3bd3a45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7fc4c3c5476f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7fc4c3c684f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7fc4c3c53226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7fc4c3c62c0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55835a27675c std::function<>::operator()() at ??:0
@ 0x7fc4c101ce06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fc4c101d6ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fc4c101f03b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55835a27675c std::function<>::operator()() at ??:0
@ 0x7fc4c100c807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fc4c184a6db start_thread at ??:0
@ 0x7fc4bed8271f clone at ??:0
I20240503 13:56:56.206279 13051 consensus_queue.cc:237] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:56.207705 13446 raft_consensus.cc:1261] T 7b57f64985b74e5fa75a5603ab8cc4bf P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Refusing update from remote peer 60ba3ffe3b7746469834cd91c84be14e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:56:56.209194 24809 consensus_queue.cc:1035] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:56:56.212111 12906 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:56:56.217432 15492 raft_consensus.cc:1261] T a8fc5c3dd305409faba2b9f1dcea0c65 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Refusing update from remote peer 009f849ee7b54c80a3f5264695047e4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:56.217532 15757 raft_consensus.cc:1261] T a8fc5c3dd305409faba2b9f1dcea0c65 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Refusing update from remote peer 009f849ee7b54c80a3f5264695047e4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:56.217916 24077 consensus_queue.cc:1035] T a8fc5c3dd305409faba2b9f1dcea0c65 P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:56.217995 24809 raft_consensus.cc:2932] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:56.219415 24077 consensus_queue.cc:1035] T a8fc5c3dd305409faba2b9f1dcea0c65 P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:56.222187 12907 catalog_manager.cc:5484] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e reported cstate change: config changed from index 4 to 5, VOTER ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) evicted. New cstate: current_term: 1 leader_uuid: "60ba3ffe3b7746469834cd91c84be14e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:56.222860 13446 raft_consensus.cc:2932] T 7b57f64985b74e5fa75a5603ab8cc4bf P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:56.226200 12905 catalog_manager.cc:5576] TS 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153) does not have the latest schema for tablet a8fc5c3dd305409faba2b9f1dcea0c65 (table rebalancer_dynamic_table [id=d2265169a02749789d7df005bb024892]). Expected version 1 got 0
I20240503 13:56:56.232715 13295 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b57f64985b74e5fa75a5603ab8cc4bf with delete_type TABLET_DATA_TOMBSTONED (TS ba0194098aa84449ae17e8a0d4c43703 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:33250
I20240503 13:56:56.232935 24523 tablet_replica.cc:330] T 7b57f64985b74e5fa75a5603ab8cc4bf P ba0194098aa84449ae17e8a0d4c43703: stopping tablet replica
I20240503 13:56:56.233083 24523 raft_consensus.cc:2229] T 7b57f64985b74e5fa75a5603ab8cc4bf P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:56.233253 24523 raft_consensus.cc:2258] T 7b57f64985b74e5fa75a5603ab8cc4bf P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:56.233784 24523 ts_tablet_manager.cc:1875] T 7b57f64985b74e5fa75a5603ab8cc4bf P ba0194098aa84449ae17e8a0d4c43703: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:56:56.236256 24523 ts_tablet_manager.cc:1888] T 7b57f64985b74e5fa75a5603ab8cc4bf P ba0194098aa84449ae17e8a0d4c43703: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 13:56:56.236323 24523 log.cc:1196] T 7b57f64985b74e5fa75a5603ab8cc4bf P ba0194098aa84449ae17e8a0d4c43703: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/wal/wals/7b57f64985b74e5fa75a5603ab8cc4bf
I20240503 13:56:56.237375 25013 tablet.cc:1722] T a8fc5c3dd305409faba2b9f1dcea0c65 P 009f849ee7b54c80a3f5264695047e4d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:56.237733 25013 tablet.cc:1620] T a8fc5c3dd305409faba2b9f1dcea0c65 P 009f849ee7b54c80a3f5264695047e4d: MemRowSet was empty: no flush needed.
I20240503 13:56:56.241261 12907 catalog_manager.cc:5576] TS 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153) does not have the latest schema for tablet a8fc5c3dd305409faba2b9f1dcea0c65 (table rebalancer_dynamic_table [id=d2265169a02749789d7df005bb024892]). Expected version 1 got 0
I20240503 13:56:56.241881 12899 catalog_manager.cc:4830] TS ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279): tablet 7b57f64985b74e5fa75a5603ab8cc4bf (table rebalance_test_table_7 [id=5e6f2f3703e1424ab61d8fe49e6266dd]) successfully deleted
I20240503 13:56:56.244148 25014 tablet.cc:1722] T a8fc5c3dd305409faba2b9f1dcea0c65 P 61b887f0d40d497badeca5604c1f0546: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:56.245123 12905 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=d2265169a02749789d7df005bb024892] alter complete (version 1)
I20240503 13:56:56.246326 23473 consensus_queue.cc:1035] T 5140b9d0611646fda51207df60c8a4df P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:56.247617 23473 consensus_queue.cc:1035] T 5140b9d0611646fda51207df60c8a4df P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:56.251240 25014 tablet.cc:1620] T a8fc5c3dd305409faba2b9f1dcea0c65 P 61b887f0d40d497badeca5604c1f0546: MemRowSet was empty: no flush needed.
I20240503 13:56:56.254297 12906 catalog_manager.cc:5576] TS 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885) does not have the latest schema for tablet a8fc5c3dd305409faba2b9f1dcea0c65 (table rebalancer_dynamic_table [id=d2265169a02749789d7df005bb024892]). Expected version 1 got 0
I20240503 13:56:56.254763 25017 tablet.cc:1722] T a8fc5c3dd305409faba2b9f1dcea0c65 P 76c386bef61a4bd5a907266a5118ffe2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:56.255100 25017 tablet.cc:1620] T a8fc5c3dd305409faba2b9f1dcea0c65 P 76c386bef61a4bd5a907266a5118ffe2: MemRowSet was empty: no flush needed.
I20240503 13:56:56.280256 12906 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_extra_table_70"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:56:56.288082 13295 tablet_service.cc:1467] Processing CreateTablet for tablet 756c5a821f1c47779af5f3009d22f1de (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=fc2ee70f451e4574a0806c7e2ce3b347]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:56.288115 15473 tablet_service.cc:1467] Processing CreateTablet for tablet 9c9942e01e89464eb9632e08c0af71c4 (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=fc2ee70f451e4574a0806c7e2ce3b347]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:56.288390 13295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 756c5a821f1c47779af5f3009d22f1de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:56.288403 15473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c9942e01e89464eb9632e08c0af71c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:56.288913 13034 tablet_service.cc:1467] Processing CreateTablet for tablet 756c5a821f1c47779af5f3009d22f1de (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=fc2ee70f451e4574a0806c7e2ce3b347]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:56.289189 13034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 756c5a821f1c47779af5f3009d22f1de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:56.291512 24796 tablet_bootstrap.cc:492] T 9c9942e01e89464eb9632e08c0af71c4 P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:56.291891 15738 tablet_service.cc:1467] Processing CreateTablet for tablet 9c9942e01e89464eb9632e08c0af71c4 (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=fc2ee70f451e4574a0806c7e2ce3b347]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:56.292157 15738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c9942e01e89464eb9632e08c0af71c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:56.292459 24796 tablet_bootstrap.cc:654] T 9c9942e01e89464eb9632e08c0af71c4 P 61b887f0d40d497badeca5604c1f0546: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:56.293267 13424 tablet_service.cc:1467] Processing CreateTablet for tablet 756c5a821f1c47779af5f3009d22f1de (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=fc2ee70f451e4574a0806c7e2ce3b347]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:56.293560 13424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 756c5a821f1c47779af5f3009d22f1de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:56.295360 13295 tablet_service.cc:1467] Processing CreateTablet for tablet 2b362fed86c042868fc5b2fbaf83b369 (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=fc2ee70f451e4574a0806c7e2ce3b347]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:56.295662 13295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b362fed86c042868fc5b2fbaf83b369. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:56.295832 25038 tablet_bootstrap.cc:492] T 756c5a821f1c47779af5f3009d22f1de P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:56.296064 15604 tablet_service.cc:1467] Processing CreateTablet for tablet 2b362fed86c042868fc5b2fbaf83b369 (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=fc2ee70f451e4574a0806c7e2ce3b347]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:56.296190 24957 tablet_bootstrap.cc:492] T 9c9942e01e89464eb9632e08c0af71c4 P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:56.296319 15604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b362fed86c042868fc5b2fbaf83b369. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:56.297041 25038 tablet_bootstrap.cc:654] T 756c5a821f1c47779af5f3009d22f1de P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:56.297041 24957 tablet_bootstrap.cc:654] T 9c9942e01e89464eb9632e08c0af71c4 P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:56.298524 24795 tablet_bootstrap.cc:492] T 2b362fed86c042868fc5b2fbaf83b369 P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:56.298803 13427 tablet_service.cc:1467] Processing CreateTablet for tablet 9c9942e01e89464eb9632e08c0af71c4 (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=fc2ee70f451e4574a0806c7e2ce3b347]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:56.299041 13427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c9942e01e89464eb9632e08c0af71c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:56.299484 24795 tablet_bootstrap.cc:654] T 2b362fed86c042868fc5b2fbaf83b369 P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:56.302095 24796 tablet_bootstrap.cc:492] T 9c9942e01e89464eb9632e08c0af71c4 P 61b887f0d40d497badeca5604c1f0546: No bootstrap required, opened a new log
I20240503 13:56:56.302209 24796 ts_tablet_manager.cc:1367] T 9c9942e01e89464eb9632e08c0af71c4 P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 13:56:56.302701 24796 raft_consensus.cc:348] T 9c9942e01e89464eb9632e08c0af71c4 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:56.302811 24796 raft_consensus.cc:374] T 9c9942e01e89464eb9632e08c0af71c4 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:56.302843 24796 raft_consensus.cc:729] T 9c9942e01e89464eb9632e08c0af71c4 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:56.302953 24796 consensus_queue.cc:260] T 9c9942e01e89464eb9632e08c0af71c4 P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:56.303256 24796 ts_tablet_manager.cc:1398] T 9c9942e01e89464eb9632e08c0af71c4 P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:56.305058 25036 tablet_bootstrap.cc:492] T 756c5a821f1c47779af5f3009d22f1de P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:56.307061 24957 tablet_bootstrap.cc:492] T 9c9942e01e89464eb9632e08c0af71c4 P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:56.307163 24957 ts_tablet_manager.cc:1367] T 9c9942e01e89464eb9632e08c0af71c4 P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 13:56:56.307664 24957 raft_consensus.cc:348] T 9c9942e01e89464eb9632e08c0af71c4 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:56.307780 24957 raft_consensus.cc:374] T 9c9942e01e89464eb9632e08c0af71c4 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:56.307811 24957 raft_consensus.cc:729] T 9c9942e01e89464eb9632e08c0af71c4 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:56.307943 24957 consensus_queue.cc:260] T 9c9942e01e89464eb9632e08c0af71c4 P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:56.308246 24957 ts_tablet_manager.cc:1398] T 9c9942e01e89464eb9632e08c0af71c4 P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:56.308610 24795 tablet_bootstrap.cc:492] T 2b362fed86c042868fc5b2fbaf83b369 P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:56.308701 24795 ts_tablet_manager.cc:1367] T 2b362fed86c042868fc5b2fbaf83b369 P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.001s
I20240503 13:56:56.309162 24795 raft_consensus.cc:348] T 2b362fed86c042868fc5b2fbaf83b369 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:56.309273 24795 raft_consensus.cc:374] T 2b362fed86c042868fc5b2fbaf83b369 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:56.309301 24795 raft_consensus.cc:729] T 2b362fed86c042868fc5b2fbaf83b369 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:56.309410 24795 consensus_queue.cc:260] T 2b362fed86c042868fc5b2fbaf83b369 P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:56.309687 24795 ts_tablet_manager.cc:1398] T 2b362fed86c042868fc5b2fbaf83b369 P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:56.325054 23094 raft_consensus.cc:482] T 9c9942e01e89464eb9632e08c0af71c4 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:56.325158 23094 raft_consensus.cc:504] T 9c9942e01e89464eb9632e08c0af71c4 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:56.325592 23094 leader_election.cc:288] T 9c9942e01e89464eb9632e08c0af71c4 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885)
I20240503 13:56:56.325974 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c9942e01e89464eb9632e08c0af71c4" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
I20240503 13:56:56.326066 25035 tablet_bootstrap.cc:492] T 756c5a821f1c47779af5f3009d22f1de P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:56.326082 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c9942e01e89464eb9632e08c0af71c4" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2" is_pre_election: true
I20240503 13:56:56.326205 15757 raft_consensus.cc:2454] T 9c9942e01e89464eb9632e08c0af71c4 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 0.
W20240503 13:56:56.326437 15426 leader_election.cc:341] T 9c9942e01e89464eb9632e08c0af71c4 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109): Illegal state: must be running to vote when last-logged opid is not known
I20240503 13:56:56.327370 15429 leader_election.cc:302] T 9c9942e01e89464eb9632e08c0af71c4 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61b887f0d40d497badeca5604c1f0546, 76c386bef61a4bd5a907266a5118ffe2; no voters: 7690ee79577c45e3a2c8f8b552c0bff8
I20240503 13:56:56.327814 20722 raft_consensus.cc:2781] T 9c9942e01e89464eb9632e08c0af71c4 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:56.327868 20722 raft_consensus.cc:482] T 9c9942e01e89464eb9632e08c0af71c4 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:56.327903 20722 raft_consensus.cc:3037] T 9c9942e01e89464eb9632e08c0af71c4 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:56.328341 25036 tablet_bootstrap.cc:654] T 756c5a821f1c47779af5f3009d22f1de P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:56.329226 20722 raft_consensus.cc:504] T 9c9942e01e89464eb9632e08c0af71c4 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:56.329382 25035 tablet_bootstrap.cc:654] T 756c5a821f1c47779af5f3009d22f1de P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:56.329627 20722 leader_election.cc:288] T 9c9942e01e89464eb9632e08c0af71c4 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 election: Requested vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885)
I20240503 13:56:56.329931 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c9942e01e89464eb9632e08c0af71c4" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
I20240503 13:56:56.330250 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c9942e01e89464eb9632e08c0af71c4" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2"
W20240503 13:56:56.330307 15426 leader_election.cc:341] T 9c9942e01e89464eb9632e08c0af71c4 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109): Illegal state: must be running to vote when last-logged opid is not known
I20240503 13:56:56.330360 15757 raft_consensus.cc:3037] T 9c9942e01e89464eb9632e08c0af71c4 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:56.330801 25038 tablet_bootstrap.cc:492] T 756c5a821f1c47779af5f3009d22f1de P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:56.330885 25038 ts_tablet_manager.cc:1367] T 756c5a821f1c47779af5f3009d22f1de P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.035s user 0.000s sys 0.002s
I20240503 13:56:56.331355 15757 raft_consensus.cc:2454] T 9c9942e01e89464eb9632e08c0af71c4 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 1.
I20240503 13:56:56.331363 25038 raft_consensus.cc:348] T 756c5a821f1c47779af5f3009d22f1de P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:56.331461 25038 raft_consensus.cc:374] T 756c5a821f1c47779af5f3009d22f1de P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:56.331502 25038 raft_consensus.cc:729] T 756c5a821f1c47779af5f3009d22f1de P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:56.331627 25038 consensus_queue.cc:260] T 756c5a821f1c47779af5f3009d22f1de P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:56.331907 25038 ts_tablet_manager.cc:1398] T 756c5a821f1c47779af5f3009d22f1de P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:56.332053 25038 tablet_bootstrap.cc:492] T 9c9942e01e89464eb9632e08c0af71c4 P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:56.332532 15429 leader_election.cc:302] T 9c9942e01e89464eb9632e08c0af71c4 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61b887f0d40d497badeca5604c1f0546, 76c386bef61a4bd5a907266a5118ffe2; no voters: 7690ee79577c45e3a2c8f8b552c0bff8
I20240503 13:56:56.333024 25038 tablet_bootstrap.cc:654] T 9c9942e01e89464eb9632e08c0af71c4 P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:56.333017 23094 raft_consensus.cc:2781] T 9c9942e01e89464eb9632e08c0af71c4 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:56.333088 23094 raft_consensus.cc:686] T 9c9942e01e89464eb9632e08c0af71c4 P 61b887f0d40d497badeca5604c1f0546 [term 1 LEADER]: Becoming Leader. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Running, Role: LEADER
I20240503 13:56:56.333216 23094 consensus_queue.cc:237] T 9c9942e01e89464eb9632e08c0af71c4 P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:56.334702 25038 tablet_bootstrap.cc:492] T 9c9942e01e89464eb9632e08c0af71c4 P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:56.334784 25038 ts_tablet_manager.cc:1367] T 9c9942e01e89464eb9632e08c0af71c4 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 13:56:56.335299 25038 raft_consensus.cc:348] T 9c9942e01e89464eb9632e08c0af71c4 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:56.335398 25038 raft_consensus.cc:374] T 9c9942e01e89464eb9632e08c0af71c4 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:56.335423 25038 raft_consensus.cc:729] T 9c9942e01e89464eb9632e08c0af71c4 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:56.335544 25038 consensus_queue.cc:260] T 9c9942e01e89464eb9632e08c0af71c4 P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:56.335808 25038 ts_tablet_manager.cc:1398] T 9c9942e01e89464eb9632e08c0af71c4 P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:56.336277 12905 catalog_manager.cc:5484] T 9c9942e01e89464eb9632e08c0af71c4 P 61b887f0d40d497badeca5604c1f0546 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61b887f0d40d497badeca5604c1f0546 (127.6.98.69). New cstate: current_term: 1 leader_uuid: "61b887f0d40d497badeca5604c1f0546" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:56.343189 25036 tablet_bootstrap.cc:492] T 756c5a821f1c47779af5f3009d22f1de P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:56.343313 25036 ts_tablet_manager.cc:1367] T 756c5a821f1c47779af5f3009d22f1de P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.038s user 0.002s sys 0.000s
I20240503 13:56:56.343931 25036 raft_consensus.cc:348] T 756c5a821f1c47779af5f3009d22f1de P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:56.344056 25036 raft_consensus.cc:374] T 756c5a821f1c47779af5f3009d22f1de P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:56.344087 25036 raft_consensus.cc:729] T 756c5a821f1c47779af5f3009d22f1de P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:56.344256 25036 consensus_queue.cc:260] T 756c5a821f1c47779af5f3009d22f1de P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:56.344635 25036 ts_tablet_manager.cc:1398] T 756c5a821f1c47779af5f3009d22f1de P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 13:56:56.345438 25035 tablet_bootstrap.cc:492] T 756c5a821f1c47779af5f3009d22f1de P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:56.345551 25035 ts_tablet_manager.cc:1367] T 756c5a821f1c47779af5f3009d22f1de P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20240503 13:56:56.346091 25035 raft_consensus.cc:348] T 756c5a821f1c47779af5f3009d22f1de P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:56.346202 25035 raft_consensus.cc:374] T 756c5a821f1c47779af5f3009d22f1de P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:56.346230 25035 raft_consensus.cc:729] T 756c5a821f1c47779af5f3009d22f1de P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:56.346372 25035 consensus_queue.cc:260] T 756c5a821f1c47779af5f3009d22f1de P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:56.346675 25035 ts_tablet_manager.cc:1398] T 756c5a821f1c47779af5f3009d22f1de P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:56.347998 25035 tablet_bootstrap.cc:492] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:56.348568 12906 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:56:56.349012 25035 tablet_bootstrap.cc:654] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:56.355494 25035 tablet_bootstrap.cc:492] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:56.355609 25035 ts_tablet_manager.cc:1367] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20240503 13:56:56.356101 25035 raft_consensus.cc:348] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:56.356220 25035 raft_consensus.cc:374] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:56.356258 25035 raft_consensus.cc:729] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:56.356415 25035 consensus_queue.cc:260] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:56.356863 25035 ts_tablet_manager.cc:1398] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240503 13:56:56.362751 25013 tablet.cc:1722] T a8fc5c3dd305409faba2b9f1dcea0c65 P 009f849ee7b54c80a3f5264695047e4d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:56.363106 25013 tablet.cc:1620] T a8fc5c3dd305409faba2b9f1dcea0c65 P 009f849ee7b54c80a3f5264695047e4d: MemRowSet was empty: no flush needed.
I20240503 13:56:56.372581 25014 tablet.cc:1722] T a8fc5c3dd305409faba2b9f1dcea0c65 P 61b887f0d40d497badeca5604c1f0546: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:56.372942 25014 tablet.cc:1620] T a8fc5c3dd305409faba2b9f1dcea0c65 P 61b887f0d40d497badeca5604c1f0546: MemRowSet was empty: no flush needed.
I20240503 13:56:56.373518 25017 tablet.cc:1722] T a8fc5c3dd305409faba2b9f1dcea0c65 P 76c386bef61a4bd5a907266a5118ffe2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:56.373811 25017 tablet.cc:1620] T a8fc5c3dd305409faba2b9f1dcea0c65 P 76c386bef61a4bd5a907266a5118ffe2: MemRowSet was empty: no flush needed.
I20240503 13:56:56.385180 12907 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=d2265169a02749789d7df005bb024892] alter complete (version 2)
I20240503 13:56:56.428720 25002 raft_consensus.cc:482] T 756c5a821f1c47779af5f3009d22f1de P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:56.428855 25002 raft_consensus.cc:504] T 756c5a821f1c47779af5f3009d22f1de P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:56.429312 25002 leader_election.cc:288] T 756c5a821f1c47779af5f3009d22f1de P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:56.430188 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "756c5a821f1c47779af5f3009d22f1de" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
I20240503 13:56:56.430188 13316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "756c5a821f1c47779af5f3009d22f1de" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703" is_pre_election: true
I20240503 13:56:56.430330 13316 raft_consensus.cc:2454] T 756c5a821f1c47779af5f3009d22f1de P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 0.
I20240503 13:56:56.430330 13445 raft_consensus.cc:2454] T 756c5a821f1c47779af5f3009d22f1de P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 0.
I20240503 13:56:56.430670 12990 leader_election.cc:302] T 756c5a821f1c47779af5f3009d22f1de P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:56.430842 25002 raft_consensus.cc:2781] T 756c5a821f1c47779af5f3009d22f1de P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:56.430892 25002 raft_consensus.cc:482] T 756c5a821f1c47779af5f3009d22f1de P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:56.430927 25002 raft_consensus.cc:3037] T 756c5a821f1c47779af5f3009d22f1de P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:56.432104 25002 raft_consensus.cc:504] T 756c5a821f1c47779af5f3009d22f1de P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:56.432480 25002 leader_election.cc:288] T 756c5a821f1c47779af5f3009d22f1de P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 election: Requested vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:56.433216 13316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "756c5a821f1c47779af5f3009d22f1de" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703"
I20240503 13:56:56.433337 13316 raft_consensus.cc:3037] T 756c5a821f1c47779af5f3009d22f1de P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:56.434396 13316 raft_consensus.cc:2454] T 756c5a821f1c47779af5f3009d22f1de P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 1.
I20240503 13:56:56.435468 12990 leader_election.cc:302] T 756c5a821f1c47779af5f3009d22f1de P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:56.435745 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "756c5a821f1c47779af5f3009d22f1de" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
I20240503 13:56:56.435865 13445 raft_consensus.cc:3037] T 756c5a821f1c47779af5f3009d22f1de P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:56.436964 13445 raft_consensus.cc:2454] T 756c5a821f1c47779af5f3009d22f1de P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 1.
I20240503 13:56:56.437474 25002 raft_consensus.cc:2781] T 756c5a821f1c47779af5f3009d22f1de P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:56.437565 25002 raft_consensus.cc:686] T 756c5a821f1c47779af5f3009d22f1de P 60ba3ffe3b7746469834cd91c84be14e [term 1 LEADER]: Becoming Leader. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Running, Role: LEADER
I20240503 13:56:56.437696 25002 consensus_queue.cc:237] T 756c5a821f1c47779af5f3009d22f1de P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:56.446167 12904 catalog_manager.cc:5484] T 756c5a821f1c47779af5f3009d22f1de P 60ba3ffe3b7746469834cd91c84be14e reported cstate change: term changed from 0 to 1, leader changed from <none> to 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65). New cstate: current_term: 1 leader_uuid: "60ba3ffe3b7746469834cd91c84be14e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
W20240503 13:56:56.467311 13380 connection.cc:535] client connection to 127.6.98.72:44753 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 13:56:56.467491 13118 connection.cc:535] client connection to 127.6.98.72:44753 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 13:56:56.467509 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 13:56:56.467501 15558 connection.cc:535] client connection to 127.6.98.72:44753 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 13:56:56.467595 13118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 13:56:56.467630 15558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 13:56:56.468029 13249 connection.cc:535] client connection to 127.6.98.72:44753 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 13:56:56.468152 13249 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 13:56:56.468240 12896 connection.cc:535] client connection to 127.6.98.72:44753 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 13:56:56.468344 12896 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 13:56:56.468575 15691 connection.cc:535] client connection to 127.6.98.72:44753 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 13:56:56.468582 12987 connection.cc:535] client connection to 127.6.98.72:44753 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 13:56:56.468675 15691 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 13:56:56.468686 12987 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 13:56:56.469303 15426 connection.cc:535] client connection to 127.6.98.72:44753 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 13:56:56.469416 15426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 13:56:56.471578 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 13:56:56.471654 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 13:56:56.471689 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 13:56:56.472061 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 13:56:56.472142 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 13:56:56.472186 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 13:56:56.472232 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 13:56:56.472267 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 13:56:56.473373 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 13:56:56.473453 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 13:56:56.473619 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 13:56:56.473773 12896 catalog_manager.cc:4573] TS e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): CreateTablet RPC failed for tablet 2b362fed86c042868fc5b2fbaf83b369: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:56:56.479851 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 13:56:56.479961 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 13:56:56.480015 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 13:56:56.480054 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 13:56:56.482560 25100 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 13:56:56.468208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.98.72:44753 (local address 127.6.98.68:42009)
0503 13:56:56.481828 (+ 13620us) negotiation.cc:107] Waiting for socket to connect
0503 13:56:56.481909 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":13522,"thread_start_us":88,"threads_started":1}
W20240503 13:56:56.482686 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 13:56:56.482734 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240503 13:56:56.482758 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 13:56:56.482784 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240503 13:56:56.482821 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection reset by peer (error 104)
W20240503 13:56:56.483273 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 13:56:56.483343 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 13:56:56.483372 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:56.486725 24869 raft_consensus.cc:482] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:56.486843 24869 raft_consensus.cc:504] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:56.487299 24869 leader_election.cc:288] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:56.488171 15623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b362fed86c042868fc5b2fbaf83b369" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d" is_pre_election: true
I20240503 13:56:56.488320 15623 raft_consensus.cc:2454] T 2b362fed86c042868fc5b2fbaf83b369 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 0.
I20240503 13:56:56.488840 13249 leader_election.cc:302] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, ba0194098aa84449ae17e8a0d4c43703; no voters:
W20240503 13:56:56.489632 13249 leader_election.cc:334] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:56:56.489889 24869 raft_consensus.cc:2781] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:56.489949 24869 raft_consensus.cc:482] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:56.489984 24869 raft_consensus.cc:3037] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:56.491204 24869 raft_consensus.cc:504] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:56.491526 24869 leader_election.cc:288] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: Requested vote from peers 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:56.491644 12906 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:56:56.491951 15623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b362fed86c042868fc5b2fbaf83b369" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d"
I20240503 13:56:56.492069 15623 raft_consensus.cc:3037] T 2b362fed86c042868fc5b2fbaf83b369 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
W20240503 13:56:56.492807 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:56.493172 15623 raft_consensus.cc:2454] T 2b362fed86c042868fc5b2fbaf83b369 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 1.
I20240503 13:56:56.493854 13249 leader_election.cc:302] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:56.494019 24869 raft_consensus.cc:2781] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election won for term 1
W20240503 13:56:56.494565 13249 leader_election.cc:334] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:56:56.497215 24869 raft_consensus.cc:686] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Becoming Leader. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Running, Role: LEADER
I20240503 13:56:56.497432 24869 consensus_queue.cc:237] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:56.501786 12904 catalog_manager.cc:5484] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 reported cstate change: term changed from 0 to 1, leader changed from <none> to ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67). New cstate: current_term: 1 leader_uuid: "ba0194098aa84449ae17e8a0d4c43703" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:56.502966 25013 tablet.cc:1722] T a8fc5c3dd305409faba2b9f1dcea0c65 P 009f849ee7b54c80a3f5264695047e4d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:56.507970 25017 tablet.cc:1722] T a8fc5c3dd305409faba2b9f1dcea0c65 P 76c386bef61a4bd5a907266a5118ffe2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:56.511796 12904 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table_new_name [id=d2265169a02749789d7df005bb024892] alter complete (version 3)
I20240503 13:56:56.511998 25014 tablet.cc:1722] T a8fc5c3dd305409faba2b9f1dcea0c65 P 61b887f0d40d497badeca5604c1f0546: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
W20240503 13:56:56.519769 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 13:56:56.525228 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 13:56:56.527601 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 13:56:56.529330 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:56.535248 15604 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 13:56:56.539498 13034 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 13:56:56.546295 13427 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240503 13:56:56.549433 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 13:56:56.550338 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 13:56:56.555802 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:56.559301 15473 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240503 13:56:56.564342 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 13:56:56.570037 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:56.573026 13163 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 13:56:56.576030 13295 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 13:56:56.576333 15738 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240503 13:56:56.582187 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 13:56:56.588958 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 13:56:56.593415 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 13:56:56.602041 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 13:56:56.603677 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 13:56:56.609890 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:56.618417 12907 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:56.618695 12907 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
W20240503 13:56:56.619650 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:56.621276 12907 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Sending DeleteTablet for 3 replicas of tablet a8fc5c3dd305409faba2b9f1dcea0c65
I20240503 13:56:56.622107 15604 tablet_service.cc:1514] Processing DeleteTablet for tablet a8fc5c3dd305409faba2b9f1dcea0c65 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:56 UTC) from {username='slave'} at 127.0.0.1:60718
I20240503 13:56:56.622642 15738 tablet_service.cc:1514] Processing DeleteTablet for tablet a8fc5c3dd305409faba2b9f1dcea0c65 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:56 UTC) from {username='slave'} at 127.0.0.1:44984
I20240503 13:56:56.622994 15473 tablet_service.cc:1514] Processing DeleteTablet for tablet a8fc5c3dd305409faba2b9f1dcea0c65 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:56 UTC) from {username='slave'} at 127.0.0.1:58758
I20240503 13:56:56.623086 25126 tablet_replica.cc:330] T a8fc5c3dd305409faba2b9f1dcea0c65 P 76c386bef61a4bd5a907266a5118ffe2: stopping tablet replica
I20240503 13:56:56.623057 25125 tablet_replica.cc:330] T a8fc5c3dd305409faba2b9f1dcea0c65 P 009f849ee7b54c80a3f5264695047e4d: stopping tablet replica
I20240503 13:56:56.623225 25126 raft_consensus.cc:2229] T a8fc5c3dd305409faba2b9f1dcea0c65 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:56.623435 25126 raft_consensus.cc:2258] T a8fc5c3dd305409faba2b9f1dcea0c65 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:56.623250 25125 raft_consensus.cc:2229] T a8fc5c3dd305409faba2b9f1dcea0c65 P 009f849ee7b54c80a3f5264695047e4d [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:56:56.623744 25125 raft_consensus.cc:2258] T a8fc5c3dd305409faba2b9f1dcea0c65 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:56.624106 25126 ts_tablet_manager.cc:1875] T a8fc5c3dd305409faba2b9f1dcea0c65 P 76c386bef61a4bd5a907266a5118ffe2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:56.624292 25125 ts_tablet_manager.cc:1875] T a8fc5c3dd305409faba2b9f1dcea0c65 P 009f849ee7b54c80a3f5264695047e4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:56.624830 25127 tablet_replica.cc:330] T a8fc5c3dd305409faba2b9f1dcea0c65 P 61b887f0d40d497badeca5604c1f0546: stopping tablet replica
I20240503 13:56:56.625041 25127 raft_consensus.cc:2229] T a8fc5c3dd305409faba2b9f1dcea0c65 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:56.625264 25127 raft_consensus.cc:2258] T a8fc5c3dd305409faba2b9f1dcea0c65 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:56.626235 25127 ts_tablet_manager.cc:1875] T a8fc5c3dd305409faba2b9f1dcea0c65 P 61b887f0d40d497badeca5604c1f0546: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:56.626927 25126 ts_tablet_manager.cc:1888] T a8fc5c3dd305409faba2b9f1dcea0c65 P 76c386bef61a4bd5a907266a5118ffe2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:56.627034 25126 log.cc:1196] T a8fc5c3dd305409faba2b9f1dcea0c65 P 76c386bef61a4bd5a907266a5118ffe2: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-6/wal/wals/a8fc5c3dd305409faba2b9f1dcea0c65
I20240503 13:56:56.627367 25126 ts_tablet_manager.cc:1909] T a8fc5c3dd305409faba2b9f1dcea0c65 P 76c386bef61a4bd5a907266a5118ffe2: Deleting consensus metadata
I20240503 13:56:56.628170 12899 catalog_manager.cc:4830] TS 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885): tablet a8fc5c3dd305409faba2b9f1dcea0c65 (table rebalancer_dynamic_table_new_name [id=d2265169a02749789d7df005bb024892]) successfully deleted
I20240503 13:56:56.628724 25127 ts_tablet_manager.cc:1888] T a8fc5c3dd305409faba2b9f1dcea0c65 P 61b887f0d40d497badeca5604c1f0546: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:56.628876 25127 log.cc:1196] T a8fc5c3dd305409faba2b9f1dcea0c65 P 61b887f0d40d497badeca5604c1f0546: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/wal/wals/a8fc5c3dd305409faba2b9f1dcea0c65
I20240503 13:56:56.629266 25127 ts_tablet_manager.cc:1909] T a8fc5c3dd305409faba2b9f1dcea0c65 P 61b887f0d40d497badeca5604c1f0546: Deleting consensus metadata
I20240503 13:56:56.630364 12899 catalog_manager.cc:4830] TS 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153): tablet a8fc5c3dd305409faba2b9f1dcea0c65 (table rebalancer_dynamic_table_new_name [id=d2265169a02749789d7df005bb024892]) successfully deleted
I20240503 13:56:56.630627 25125 ts_tablet_manager.cc:1888] T a8fc5c3dd305409faba2b9f1dcea0c65 P 009f849ee7b54c80a3f5264695047e4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:56.630765 25125 log.cc:1196] T a8fc5c3dd305409faba2b9f1dcea0c65 P 009f849ee7b54c80a3f5264695047e4d: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-5/wal/wals/a8fc5c3dd305409faba2b9f1dcea0c65
I20240503 13:56:56.631079 25125 ts_tablet_manager.cc:1909] T a8fc5c3dd305409faba2b9f1dcea0c65 P 009f849ee7b54c80a3f5264695047e4d: Deleting consensus metadata
I20240503 13:56:56.632921 12896 catalog_manager.cc:4830] TS 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951): tablet a8fc5c3dd305409faba2b9f1dcea0c65 (table rebalancer_dynamic_table_new_name [id=d2265169a02749789d7df005bb024892]) successfully deleted
I20240503 13:56:56.634658 25056 raft_consensus.cc:482] T 9c9942e01e89464eb9632e08c0af71c4 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:56.634768 25056 raft_consensus.cc:504] T 9c9942e01e89464eb9632e08c0af71c4 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:56.635210 25056 leader_election.cc:288] T 9c9942e01e89464eb9632e08c0af71c4 P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153), 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885)
I20240503 13:56:56.635744 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c9942e01e89464eb9632e08c0af71c4" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2" is_pre_election: true
I20240503 13:56:56.635843 15492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c9942e01e89464eb9632e08c0af71c4" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546" is_pre_election: true
I20240503 13:56:56.635916 15757 raft_consensus.cc:2379] T 9c9942e01e89464eb9632e08c0af71c4 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7690ee79577c45e3a2c8f8b552c0bff8 in current term 1: Already voted for candidate 61b887f0d40d497badeca5604c1f0546 in this term.
I20240503 13:56:56.636242 13383 leader_election.cc:302] T 9c9942e01e89464eb9632e08c0af71c4 P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8; no voters: 61b887f0d40d497badeca5604c1f0546, 76c386bef61a4bd5a907266a5118ffe2
I20240503 13:56:56.636546 25056 raft_consensus.cc:3037] T 9c9942e01e89464eb9632e08c0af71c4 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:56.637665 25056 raft_consensus.cc:2726] T 9c9942e01e89464eb9632e08c0af71c4 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 13:56:56.672392 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:56:56.682258 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 13:56:56.699328 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:56.723783 12904 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:56:56.738795 13295 tablet_service.cc:1467] Processing CreateTablet for tablet 2427783ee0e348b98453818704ece5f8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2ad9a0e46b12442cad28818be33337b2]), partition=
I20240503 13:56:56.738988 15473 tablet_service.cc:1467] Processing CreateTablet for tablet 2427783ee0e348b98453818704ece5f8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2ad9a0e46b12442cad28818be33337b2]), partition=
I20240503 13:56:56.739090 13295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2427783ee0e348b98453818704ece5f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:56.739213 15473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2427783ee0e348b98453818704ece5f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:56.741497 25035 tablet_bootstrap.cc:492] T 2427783ee0e348b98453818704ece5f8 P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:56.742385 25035 tablet_bootstrap.cc:654] T 2427783ee0e348b98453818704ece5f8 P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:56.743203 24796 tablet_bootstrap.cc:492] T 2427783ee0e348b98453818704ece5f8 P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:56.744102 24796 tablet_bootstrap.cc:654] T 2427783ee0e348b98453818704ece5f8 P 61b887f0d40d497badeca5604c1f0546: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:56.753917 24796 tablet_bootstrap.cc:492] T 2427783ee0e348b98453818704ece5f8 P 61b887f0d40d497badeca5604c1f0546: No bootstrap required, opened a new log
I20240503 13:56:56.754035 24796 ts_tablet_manager.cc:1367] T 2427783ee0e348b98453818704ece5f8 P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 13:56:56.754565 24796 raft_consensus.cc:348] T 2427783ee0e348b98453818704ece5f8 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:56.754685 24796 raft_consensus.cc:374] T 2427783ee0e348b98453818704ece5f8 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:56.754719 24796 raft_consensus.cc:729] T 2427783ee0e348b98453818704ece5f8 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:56.754839 24796 consensus_queue.cc:260] T 2427783ee0e348b98453818704ece5f8 P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:56.755174 24796 ts_tablet_manager.cc:1398] T 2427783ee0e348b98453818704ece5f8 P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:56.757828 25035 tablet_bootstrap.cc:492] T 2427783ee0e348b98453818704ece5f8 P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:56.757941 25035 ts_tablet_manager.cc:1367] T 2427783ee0e348b98453818704ece5f8 P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.001s
I20240503 13:56:56.758383 25035 raft_consensus.cc:348] T 2427783ee0e348b98453818704ece5f8 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:56.758517 25035 raft_consensus.cc:374] T 2427783ee0e348b98453818704ece5f8 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:56.758553 25035 raft_consensus.cc:729] T 2427783ee0e348b98453818704ece5f8 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:56.758683 25035 consensus_queue.cc:260] T 2427783ee0e348b98453818704ece5f8 P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:56.758965 25035 ts_tablet_manager.cc:1398] T 2427783ee0e348b98453818704ece5f8 P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:56.790828 15473 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240503 13:56:56.793646 13034 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 13:56:56.803555 15604 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 13:56:56.805925 13427 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 13:56:56.806967 13163 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 13:56:56.808881 13295 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 13:56:56.817246 15738 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 13:56:56.831261 23094 consensus_queue.cc:1035] T 9c9942e01e89464eb9632e08c0af71c4 P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:56.858880 25096 consensus_queue.cc:1035] T 9c9942e01e89464eb9632e08c0af71c4 P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:56.916538 24809 consensus_queue.cc:1035] T 756c5a821f1c47779af5f3009d22f1de P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:56.920342 24809 consensus_queue.cc:1035] T 756c5a821f1c47779af5f3009d22f1de P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:56.938565 24869 raft_consensus.cc:482] T 2427783ee0e348b98453818704ece5f8 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:56.938705 24869 raft_consensus.cc:504] T 2427783ee0e348b98453818704ece5f8 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:56.939087 24869 leader_election.cc:288] T 2427783ee0e348b98453818704ece5f8 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:56.939420 15493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2427783ee0e348b98453818704ece5f8" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546" is_pre_election: true
I20240503 13:56:56.939567 15493 raft_consensus.cc:2454] T 2427783ee0e348b98453818704ece5f8 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 0.
I20240503 13:56:56.939852 13252 leader_election.cc:302] T 2427783ee0e348b98453818704ece5f8 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61b887f0d40d497badeca5604c1f0546, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:56.940008 24869 raft_consensus.cc:2781] T 2427783ee0e348b98453818704ece5f8 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:56.940057 24869 raft_consensus.cc:482] T 2427783ee0e348b98453818704ece5f8 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:56.940094 24869 raft_consensus.cc:3037] T 2427783ee0e348b98453818704ece5f8 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
W20240503 13:56:56.940229 13249 leader_election.cc:334] T 2427783ee0e348b98453818704ece5f8 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:56:56.941141 24869 raft_consensus.cc:504] T 2427783ee0e348b98453818704ece5f8 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:56.941474 24869 leader_election.cc:288] T 2427783ee0e348b98453818704ece5f8 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: Requested vote from peers 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:56.942046 15493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2427783ee0e348b98453818704ece5f8" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546"
I20240503 13:56:56.942153 15493 raft_consensus.cc:3037] T 2427783ee0e348b98453818704ece5f8 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Advancing to term 1
W20240503 13:56:56.942374 13249 leader_election.cc:334] T 2427783ee0e348b98453818704ece5f8 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:56:56.943114 15493 raft_consensus.cc:2454] T 2427783ee0e348b98453818704ece5f8 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 1.
I20240503 13:56:56.943532 13252 leader_election.cc:302] T 2427783ee0e348b98453818704ece5f8 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61b887f0d40d497badeca5604c1f0546, ba0194098aa84449ae17e8a0d4c43703; no voters: e47105a0f46e4511b7112e571688a4c7
I20240503 13:56:56.943766 24869 raft_consensus.cc:2781] T 2427783ee0e348b98453818704ece5f8 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:56.944085 24869 raft_consensus.cc:686] T 2427783ee0e348b98453818704ece5f8 P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Becoming Leader. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Running, Role: LEADER
I20240503 13:56:56.944236 24869 consensus_queue.cc:237] T 2427783ee0e348b98453818704ece5f8 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
W20240503 13:56:56.945041 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:56.946581 12904 catalog_manager.cc:5484] T 2427783ee0e348b98453818704ece5f8 P ba0194098aa84449ae17e8a0d4c43703 reported cstate change: term changed from 0 to 1, leader changed from <none> to ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67). New cstate: current_term: 1 leader_uuid: "ba0194098aa84449ae17e8a0d4c43703" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:56.975456 24869 consensus_queue.cc:1035] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:57.012811 12904 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_extra_table_72"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:56:57.019100 13034 tablet_service.cc:1467] Processing CreateTablet for tablet 67395f0e6df94bd49aa76359858ad327 (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=654c7f8ecc3d4ec79f279917776c8874]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:57.019290 13295 tablet_service.cc:1467] Processing CreateTablet for tablet 67395f0e6df94bd49aa76359858ad327 (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=654c7f8ecc3d4ec79f279917776c8874]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:57.019438 13034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67395f0e6df94bd49aa76359858ad327. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:57.019553 13295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67395f0e6df94bd49aa76359858ad327. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:57.020038 15604 tablet_service.cc:1467] Processing CreateTablet for tablet 6483eb56a18c41979d1faac1ef0b7816 (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=654c7f8ecc3d4ec79f279917776c8874]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:57.020278 15604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6483eb56a18c41979d1faac1ef0b7816. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:57.021325 25196 tablet_bootstrap.cc:492] T 67395f0e6df94bd49aa76359858ad327 P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:57.021724 25035 tablet_bootstrap.cc:492] T 67395f0e6df94bd49aa76359858ad327 P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:57.022159 13163 tablet_service.cc:1467] Processing CreateTablet for tablet ff21b7dd34b54f5f92a8300e81bc5b1c (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=654c7f8ecc3d4ec79f279917776c8874]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:57.022313 25196 tablet_bootstrap.cc:654] T 67395f0e6df94bd49aa76359858ad327 P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:57.022409 13163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff21b7dd34b54f5f92a8300e81bc5b1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:57.022614 25035 tablet_bootstrap.cc:654] T 67395f0e6df94bd49aa76359858ad327 P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:57.023382 13164 tablet_service.cc:1467] Processing CreateTablet for tablet 6483eb56a18c41979d1faac1ef0b7816 (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=654c7f8ecc3d4ec79f279917776c8874]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:57.023576 25197 tablet_bootstrap.cc:492] T 6483eb56a18c41979d1faac1ef0b7816 P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:57.023610 13164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6483eb56a18c41979d1faac1ef0b7816. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:57.024470 25200 tablet_bootstrap.cc:492] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:57.024471 25197 tablet_bootstrap.cc:654] T 6483eb56a18c41979d1faac1ef0b7816 P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:57.025432 15738 tablet_service.cc:1467] Processing CreateTablet for tablet 67395f0e6df94bd49aa76359858ad327 (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=654c7f8ecc3d4ec79f279917776c8874]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:57.025696 15738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67395f0e6df94bd49aa76359858ad327. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:57.025784 25197 tablet_bootstrap.cc:492] T 6483eb56a18c41979d1faac1ef0b7816 P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:57.025866 25197 ts_tablet_manager.cc:1367] T 6483eb56a18c41979d1faac1ef0b7816 P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:56:57.026298 25197 raft_consensus.cc:348] T 6483eb56a18c41979d1faac1ef0b7816 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:57.026414 25197 raft_consensus.cc:374] T 6483eb56a18c41979d1faac1ef0b7816 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:57.026445 25197 raft_consensus.cc:729] T 6483eb56a18c41979d1faac1ef0b7816 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:57.026553 25197 consensus_queue.cc:260] T 6483eb56a18c41979d1faac1ef0b7816 P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:57.026587 13295 tablet_service.cc:1467] Processing CreateTablet for tablet 6483eb56a18c41979d1faac1ef0b7816 (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=654c7f8ecc3d4ec79f279917776c8874]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:57.026677 13034 tablet_service.cc:1467] Processing CreateTablet for tablet ff21b7dd34b54f5f92a8300e81bc5b1c (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=654c7f8ecc3d4ec79f279917776c8874]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:57.026799 13295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6483eb56a18c41979d1faac1ef0b7816. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:57.026818 25197 ts_tablet_manager.cc:1398] T 6483eb56a18c41979d1faac1ef0b7816 P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:57.026918 13034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff21b7dd34b54f5f92a8300e81bc5b1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:57.027892 25202 tablet_bootstrap.cc:492] T 67395f0e6df94bd49aa76359858ad327 P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:57.029115 25200 tablet_bootstrap.cc:654] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:57.029490 25202 tablet_bootstrap.cc:654] T 67395f0e6df94bd49aa76359858ad327 P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:57.031608 25035 tablet_bootstrap.cc:492] T 67395f0e6df94bd49aa76359858ad327 P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:57.031702 25035 ts_tablet_manager.cc:1367] T 67395f0e6df94bd49aa76359858ad327 P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.001s
I20240503 13:56:57.032326 25035 raft_consensus.cc:348] T 67395f0e6df94bd49aa76359858ad327 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:57.032442 25196 tablet_bootstrap.cc:492] T 67395f0e6df94bd49aa76359858ad327 P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:57.032449 25035 raft_consensus.cc:374] T 67395f0e6df94bd49aa76359858ad327 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:57.032500 25035 raft_consensus.cc:729] T 67395f0e6df94bd49aa76359858ad327 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:57.032624 25035 consensus_queue.cc:260] T 67395f0e6df94bd49aa76359858ad327 P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:57.032559 25196 ts_tablet_manager.cc:1367] T 67395f0e6df94bd49aa76359858ad327 P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 13:56:57.032917 25035 ts_tablet_manager.cc:1398] T 67395f0e6df94bd49aa76359858ad327 P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:57.033437 25035 tablet_bootstrap.cc:492] T 6483eb56a18c41979d1faac1ef0b7816 P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:57.033423 25196 raft_consensus.cc:348] T 67395f0e6df94bd49aa76359858ad327 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:57.033519 25196 raft_consensus.cc:374] T 67395f0e6df94bd49aa76359858ad327 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:57.033555 25196 raft_consensus.cc:729] T 67395f0e6df94bd49aa76359858ad327 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:57.033694 25196 consensus_queue.cc:260] T 67395f0e6df94bd49aa76359858ad327 P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:57.034011 25196 ts_tablet_manager.cc:1398] T 67395f0e6df94bd49aa76359858ad327 P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:57.034193 25196 tablet_bootstrap.cc:492] T ff21b7dd34b54f5f92a8300e81bc5b1c P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
I20240503 13:56:57.034399 25035 tablet_bootstrap.cc:654] T 6483eb56a18c41979d1faac1ef0b7816 P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:57.035068 25196 tablet_bootstrap.cc:654] T ff21b7dd34b54f5f92a8300e81bc5b1c P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:57.035933 25200 tablet_bootstrap.cc:492] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:57.036026 25200 ts_tablet_manager.cc:1367] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.002s
I20240503 13:56:57.036110 25035 tablet_bootstrap.cc:492] T 6483eb56a18c41979d1faac1ef0b7816 P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:57.036178 25035 ts_tablet_manager.cc:1367] T 6483eb56a18c41979d1faac1ef0b7816 P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:56:57.036517 25200 raft_consensus.cc:348] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:57.036617 25200 raft_consensus.cc:374] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:57.036648 25200 raft_consensus.cc:729] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:57.036664 25035 raft_consensus.cc:348] T 6483eb56a18c41979d1faac1ef0b7816 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:57.036746 25035 raft_consensus.cc:374] T 6483eb56a18c41979d1faac1ef0b7816 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:57.036773 25035 raft_consensus.cc:729] T 6483eb56a18c41979d1faac1ef0b7816 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:57.036787 25200 consensus_queue.cc:260] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:57.036868 25035 consensus_queue.cc:260] T 6483eb56a18c41979d1faac1ef0b7816 P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:57.037051 25200 ts_tablet_manager.cc:1398] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:56:57.037123 25035 ts_tablet_manager.cc:1398] T 6483eb56a18c41979d1faac1ef0b7816 P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:57.037987 25200 tablet_bootstrap.cc:492] T 6483eb56a18c41979d1faac1ef0b7816 P 8ccf5a4ac450414c9d3579bd5800b770: Bootstrap starting.
I20240503 13:56:57.038872 25200 tablet_bootstrap.cc:654] T 6483eb56a18c41979d1faac1ef0b7816 P 8ccf5a4ac450414c9d3579bd5800b770: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:57.040213 25202 tablet_bootstrap.cc:492] T 67395f0e6df94bd49aa76359858ad327 P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:57.040612 25202 ts_tablet_manager.cc:1367] T 67395f0e6df94bd49aa76359858ad327 P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 13:56:57.040843 25196 tablet_bootstrap.cc:492] T ff21b7dd34b54f5f92a8300e81bc5b1c P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:57.040931 25196 ts_tablet_manager.cc:1367] T ff21b7dd34b54f5f92a8300e81bc5b1c P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 13:56:57.041342 25196 raft_consensus.cc:348] T ff21b7dd34b54f5f92a8300e81bc5b1c P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:57.041451 25196 raft_consensus.cc:374] T ff21b7dd34b54f5f92a8300e81bc5b1c P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:57.041479 25196 raft_consensus.cc:729] T ff21b7dd34b54f5f92a8300e81bc5b1c P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:57.041581 25196 consensus_queue.cc:260] T ff21b7dd34b54f5f92a8300e81bc5b1c P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:57.042351 25196 ts_tablet_manager.cc:1398] T ff21b7dd34b54f5f92a8300e81bc5b1c P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:57.042613 25202 raft_consensus.cc:348] T 67395f0e6df94bd49aa76359858ad327 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:57.042785 25202 raft_consensus.cc:374] T 67395f0e6df94bd49aa76359858ad327 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:57.042923 25202 raft_consensus.cc:729] T 67395f0e6df94bd49aa76359858ad327 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:57.043126 25202 consensus_queue.cc:260] T 67395f0e6df94bd49aa76359858ad327 P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:57.043407 25202 ts_tablet_manager.cc:1398] T 67395f0e6df94bd49aa76359858ad327 P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:56:57.045181 25200 tablet_bootstrap.cc:492] T 6483eb56a18c41979d1faac1ef0b7816 P 8ccf5a4ac450414c9d3579bd5800b770: No bootstrap required, opened a new log
I20240503 13:56:57.045336 25200 ts_tablet_manager.cc:1367] T 6483eb56a18c41979d1faac1ef0b7816 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240503 13:56:57.046821 25200 raft_consensus.cc:348] T 6483eb56a18c41979d1faac1ef0b7816 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:57.047042 25200 raft_consensus.cc:374] T 6483eb56a18c41979d1faac1ef0b7816 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:57.047124 25200 raft_consensus.cc:729] T 6483eb56a18c41979d1faac1ef0b7816 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Initialized, Role: FOLLOWER
I20240503 13:56:57.047318 25200 consensus_queue.cc:260] T 6483eb56a18c41979d1faac1ef0b7816 P 8ccf5a4ac450414c9d3579bd5800b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:57.048640 25200 ts_tablet_manager.cc:1398] T 6483eb56a18c41979d1faac1ef0b7816 P 8ccf5a4ac450414c9d3579bd5800b770: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 13:56:57.054970 12910 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:56:57.058916 15492 raft_consensus.cc:1261] T 2427783ee0e348b98453818704ece5f8 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Refusing update from remote peer ba0194098aa84449ae17e8a0d4c43703: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:57.059387 24869 consensus_queue.cc:1035] T 2427783ee0e348b98453818704ece5f8 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 13:56:57.059546 13249 consensus_peers.cc:485] T 2427783ee0e348b98453818704ece5f8 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:57.062263 25210 tablet.cc:1722] T 2427783ee0e348b98453818704ece5f8 P 61b887f0d40d497badeca5604c1f0546: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:57.062546 12910 catalog_manager.cc:5576] TS 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153) does not have the latest schema for tablet 2427783ee0e348b98453818704ece5f8 (table rebalancer_dynamic_table [id=2ad9a0e46b12442cad28818be33337b2]). Expected version 1 got 0
I20240503 13:56:57.062595 25210 tablet.cc:1620] T 2427783ee0e348b98453818704ece5f8 P 61b887f0d40d497badeca5604c1f0546: MemRowSet was empty: no flush needed.
I20240503 13:56:57.062968 25209 tablet.cc:1722] T 2427783ee0e348b98453818704ece5f8 P ba0194098aa84449ae17e8a0d4c43703: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:57.063987 25209 tablet.cc:1620] T 2427783ee0e348b98453818704ece5f8 P ba0194098aa84449ae17e8a0d4c43703: MemRowSet was empty: no flush needed.
I20240503 13:56:57.066879 12910 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=2ad9a0e46b12442cad28818be33337b2] alter complete (version 1)
I20240503 13:56:57.089090 24831 raft_consensus.cc:482] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:57.089226 24831 raft_consensus.cc:504] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:57.089633 24831 leader_election.cc:288] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:57.090147 13051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff21b7dd34b54f5f92a8300e81bc5b1c" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e" is_pre_election: true
I20240503 13:56:57.090288 13051 raft_consensus.cc:2454] T ff21b7dd34b54f5f92a8300e81bc5b1c P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 0.
I20240503 13:56:57.090620 13119 leader_election.cc:302] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 8ccf5a4ac450414c9d3579bd5800b770; no voters:
W20240503 13:56:57.090711 13118 leader_election.cc:334] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:56:57.090866 24831 raft_consensus.cc:2781] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:57.090919 24831 raft_consensus.cc:482] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:57.090963 24831 raft_consensus.cc:3037] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:57.092044 24831 raft_consensus.cc:504] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:57.092377 24831 leader_election.cc:288] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 election: Requested vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:57.093047 13051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff21b7dd34b54f5f92a8300e81bc5b1c" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e"
I20240503 13:56:57.093175 13051 raft_consensus.cc:3037] T ff21b7dd34b54f5f92a8300e81bc5b1c P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:57.094194 13051 raft_consensus.cc:2454] T ff21b7dd34b54f5f92a8300e81bc5b1c P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 1.
I20240503 13:56:57.094465 13119 leader_election.cc:302] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 8ccf5a4ac450414c9d3579bd5800b770; no voters:
I20240503 13:56:57.094630 24831 raft_consensus.cc:2781] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:57.094698 24831 raft_consensus.cc:686] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 LEADER]: Becoming Leader. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Running, Role: LEADER
I20240503 13:56:57.094830 24831 consensus_queue.cc:237] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:57.097290 12908 catalog_manager.cc:5484] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66). New cstate: current_term: 1 leader_uuid: "8ccf5a4ac450414c9d3579bd5800b770" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } }
W20240503 13:56:57.097997 13118 leader_election.cc:334] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:56:57.115725 24869 raft_consensus.cc:482] T 67395f0e6df94bd49aa76359858ad327 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:57.115849 24869 raft_consensus.cc:504] T 67395f0e6df94bd49aa76359858ad327 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:57.116305 24869 leader_election.cc:288] T 67395f0e6df94bd49aa76359858ad327 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695), 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885)
I20240503 13:56:57.116756 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67395f0e6df94bd49aa76359858ad327" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2" is_pre_election: true
I20240503 13:56:57.116756 13051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67395f0e6df94bd49aa76359858ad327" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e" is_pre_election: true
I20240503 13:56:57.116887 15757 raft_consensus.cc:2454] T 67395f0e6df94bd49aa76359858ad327 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 0.
I20240503 13:56:57.116887 13051 raft_consensus.cc:2454] T 67395f0e6df94bd49aa76359858ad327 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 0.
I20240503 13:56:57.117149 13250 leader_election.cc:302] T 67395f0e6df94bd49aa76359858ad327 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:57.117429 24869 raft_consensus.cc:2781] T 67395f0e6df94bd49aa76359858ad327 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:57.117559 24869 raft_consensus.cc:482] T 67395f0e6df94bd49aa76359858ad327 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:57.117642 24869 raft_consensus.cc:3037] T 67395f0e6df94bd49aa76359858ad327 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:57.118628 24869 raft_consensus.cc:504] T 67395f0e6df94bd49aa76359858ad327 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:57.118995 24869 leader_election.cc:288] T 67395f0e6df94bd49aa76359858ad327 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: Requested vote from peers 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695), 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885)
I20240503 13:56:57.119323 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67395f0e6df94bd49aa76359858ad327" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2"
I20240503 13:56:57.119385 13051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67395f0e6df94bd49aa76359858ad327" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e"
I20240503 13:56:57.119441 15757 raft_consensus.cc:3037] T 67395f0e6df94bd49aa76359858ad327 P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:57.119503 13051 raft_consensus.cc:3037] T 67395f0e6df94bd49aa76359858ad327 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:57.120520 13051 raft_consensus.cc:2454] T 67395f0e6df94bd49aa76359858ad327 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 1.
I20240503 13:56:57.120575 15757 raft_consensus.cc:2454] T 67395f0e6df94bd49aa76359858ad327 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 1.
I20240503 13:56:57.120895 13250 leader_election.cc:302] T 67395f0e6df94bd49aa76359858ad327 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:57.121397 24869 raft_consensus.cc:2781] T 67395f0e6df94bd49aa76359858ad327 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:57.121496 24869 raft_consensus.cc:686] T 67395f0e6df94bd49aa76359858ad327 P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Becoming Leader. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Running, Role: LEADER
I20240503 13:56:57.121624 24869 consensus_queue.cc:237] T 67395f0e6df94bd49aa76359858ad327 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:57.124145 12908 catalog_manager.cc:5484] T 67395f0e6df94bd49aa76359858ad327 P ba0194098aa84449ae17e8a0d4c43703 reported cstate change: term changed from 0 to 1, leader changed from <none> to ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67). New cstate: current_term: 1 leader_uuid: "ba0194098aa84449ae17e8a0d4c43703" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:57.169035 12912 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:56:57.176513 25209 tablet.cc:1722] T 2427783ee0e348b98453818704ece5f8 P ba0194098aa84449ae17e8a0d4c43703: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:57.176849 25209 tablet.cc:1620] T 2427783ee0e348b98453818704ece5f8 P ba0194098aa84449ae17e8a0d4c43703: MemRowSet was empty: no flush needed.
I20240503 13:56:57.179152 25210 tablet.cc:1722] T 2427783ee0e348b98453818704ece5f8 P 61b887f0d40d497badeca5604c1f0546: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:57.179556 25210 tablet.cc:1620] T 2427783ee0e348b98453818704ece5f8 P 61b887f0d40d497badeca5604c1f0546: MemRowSet was empty: no flush needed.
W20240503 13:56:57.192159 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:56:57.200091 12906 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=2ad9a0e46b12442cad28818be33337b2] alter complete (version 2)
I20240503 13:56:57.204566 23473 raft_consensus.cc:482] T 6483eb56a18c41979d1faac1ef0b7816 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:57.204703 23473 raft_consensus.cc:504] T 6483eb56a18c41979d1faac1ef0b7816 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:57.205042 23473 leader_election.cc:288] T 6483eb56a18c41979d1faac1ef0b7816 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:57.205621 15623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6483eb56a18c41979d1faac1ef0b7816" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d" is_pre_election: true
I20240503 13:56:57.205749 15623 raft_consensus.cc:2454] T 6483eb56a18c41979d1faac1ef0b7816 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 0.
I20240503 13:56:57.205930 13183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6483eb56a18c41979d1faac1ef0b7816" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770" is_pre_election: true
I20240503 13:56:57.206055 13249 leader_election.cc:302] T 6483eb56a18c41979d1faac1ef0b7816 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:57.206079 13183 raft_consensus.cc:2454] T 6483eb56a18c41979d1faac1ef0b7816 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 0.
I20240503 13:56:57.206200 23473 raft_consensus.cc:2781] T 6483eb56a18c41979d1faac1ef0b7816 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:57.206254 23473 raft_consensus.cc:482] T 6483eb56a18c41979d1faac1ef0b7816 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:57.206293 23473 raft_consensus.cc:3037] T 6483eb56a18c41979d1faac1ef0b7816 P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:57.207428 23473 raft_consensus.cc:504] T 6483eb56a18c41979d1faac1ef0b7816 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:57.207742 23473 leader_election.cc:288] T 6483eb56a18c41979d1faac1ef0b7816 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: Requested vote from peers 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:57.211525 13183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6483eb56a18c41979d1faac1ef0b7816" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770"
I20240503 13:56:57.211665 13183 raft_consensus.cc:3037] T 6483eb56a18c41979d1faac1ef0b7816 P 8ccf5a4ac450414c9d3579bd5800b770 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:57.212745 13183 raft_consensus.cc:2454] T 6483eb56a18c41979d1faac1ef0b7816 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 1.
I20240503 13:56:57.213675 15623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6483eb56a18c41979d1faac1ef0b7816" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d"
I20240503 13:56:57.213801 15623 raft_consensus.cc:3037] T 6483eb56a18c41979d1faac1ef0b7816 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:57.214864 15623 raft_consensus.cc:2454] T 6483eb56a18c41979d1faac1ef0b7816 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 1.
I20240503 13:56:57.215415 13252 leader_election.cc:302] T 6483eb56a18c41979d1faac1ef0b7816 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ccf5a4ac450414c9d3579bd5800b770, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:57.216581 23473 raft_consensus.cc:2781] T 6483eb56a18c41979d1faac1ef0b7816 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:57.216681 23473 raft_consensus.cc:686] T 6483eb56a18c41979d1faac1ef0b7816 P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Becoming Leader. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Running, Role: LEADER
I20240503 13:56:57.216830 23473 consensus_queue.cc:237] T 6483eb56a18c41979d1faac1ef0b7816 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:57.223325 12904 catalog_manager.cc:5484] T 6483eb56a18c41979d1faac1ef0b7816 P ba0194098aa84449ae17e8a0d4c43703 reported cstate change: term changed from 0 to 1, leader changed from <none> to ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67). New cstate: current_term: 1 leader_uuid: "ba0194098aa84449ae17e8a0d4c43703" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } }
I20240503 13:56:57.232612 24959 raft_consensus.cc:482] T 25d41dbdacc341ee93c86f7a7f056123 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e47105a0f46e4511b7112e571688a4c7)
I20240503 13:56:57.232719 24959 raft_consensus.cc:504] T 25d41dbdacc341ee93c86f7a7f056123 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:57.233144 24959 leader_election.cc:288] T 25d41dbdacc341ee93c86f7a7f056123 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:57.233736 15492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25d41dbdacc341ee93c86f7a7f056123" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546" is_pre_election: true
W20240503 13:56:57.234552 15691 leader_election.cc:334] T 25d41dbdacc341ee93c86f7a7f056123 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:56:57.234627 15691 leader_election.cc:302] T 25d41dbdacc341ee93c86f7a7f056123 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76c386bef61a4bd5a907266a5118ffe2; no voters: 61b887f0d40d497badeca5604c1f0546, e47105a0f46e4511b7112e571688a4c7
I20240503 13:56:57.235164 24959 raft_consensus.cc:2726] T 25d41dbdacc341ee93c86f7a7f056123 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 13:56:57.244165 24637 raft_consensus.cc:482] T f58fc3bc579748bb88de34612808ccda P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e47105a0f46e4511b7112e571688a4c7)
I20240503 13:56:57.244275 24637 raft_consensus.cc:504] T f58fc3bc579748bb88de34612808ccda P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:57.244663 24637 leader_election.cc:288] T f58fc3bc579748bb88de34612808ccda P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:57.245163 13052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f58fc3bc579748bb88de34612808ccda" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e" is_pre_election: true
W20240503 13:56:57.246429 13380 leader_election.cc:334] T f58fc3bc579748bb88de34612808ccda P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:56:57.246490 13380 leader_election.cc:302] T f58fc3bc579748bb88de34612808ccda P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8; no voters: 60ba3ffe3b7746469834cd91c84be14e, e47105a0f46e4511b7112e571688a4c7
I20240503 13:56:57.246753 24637 raft_consensus.cc:2726] T f58fc3bc579748bb88de34612808ccda P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 13:56:57.248634 13034 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 13:56:57.248910 15604 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 13:56:57.255735 15473 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240503 13:56:57.258105 24901 raft_consensus.cc:482] T d3849e4effe64656bd8b0f504c9dc17a P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e47105a0f46e4511b7112e571688a4c7)
I20240503 13:56:57.258215 24901 raft_consensus.cc:504] T d3849e4effe64656bd8b0f504c9dc17a P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:57.258616 24901 leader_election.cc:288] T d3849e4effe64656bd8b0f504c9dc17a P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:57.259306 13183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3849e4effe64656bd8b0f504c9dc17a" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770" is_pre_election: true
W20240503 13:56:57.259665 15558 leader_election.cc:334] T d3849e4effe64656bd8b0f504c9dc17a P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:56:57.259644 25254 raft_consensus.cc:482] T 557f14aaf5564ebaa461720a4d7997ff P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e47105a0f46e4511b7112e571688a4c7)
I20240503 13:56:57.259739 25254 raft_consensus.cc:504] T 557f14aaf5564ebaa461720a4d7997ff P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:57.260083 25254 leader_election.cc:288] T 557f14aaf5564ebaa461720a4d7997ff P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951)
I20240503 13:56:57.260929 15624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "557f14aaf5564ebaa461720a4d7997ff" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d" is_pre_election: true
I20240503 13:56:57.261740 15561 leader_election.cc:302] T d3849e4effe64656bd8b0f504c9dc17a P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d; no voters: 8ccf5a4ac450414c9d3579bd5800b770, e47105a0f46e4511b7112e571688a4c7
I20240503 13:56:57.262115 23621 raft_consensus.cc:2726] T d3849e4effe64656bd8b0f504c9dc17a P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 13:56:57.262567 24369 raft_consensus.cc:482] T 10ee7dee14364389871900670a02906a P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e47105a0f46e4511b7112e571688a4c7)
I20240503 13:56:57.262660 24369 raft_consensus.cc:504] T 10ee7dee14364389871900670a02906a P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:57.262977 24369 leader_election.cc:288] T 10ee7dee14364389871900670a02906a P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:57.263661 15624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10ee7dee14364389871900670a02906a" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d" is_pre_election: true
W20240503 13:56:57.264189 13249 leader_election.cc:334] T 557f14aaf5564ebaa461720a4d7997ff P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:56:57.264256 13249 leader_election.cc:302] T 557f14aaf5564ebaa461720a4d7997ff P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba0194098aa84449ae17e8a0d4c43703; no voters: 009f849ee7b54c80a3f5264695047e4d, e47105a0f46e4511b7112e571688a4c7
I20240503 13:56:57.264456 25254 raft_consensus.cc:2726] T 557f14aaf5564ebaa461720a4d7997ff P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 13:56:57.264716 15691 leader_election.cc:334] T 10ee7dee14364389871900670a02906a P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:56:57.264779 15691 leader_election.cc:302] T 10ee7dee14364389871900670a02906a P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76c386bef61a4bd5a907266a5118ffe2; no voters: 009f849ee7b54c80a3f5264695047e4d, e47105a0f46e4511b7112e571688a4c7
I20240503 13:56:57.265005 24959 raft_consensus.cc:2726] T 10ee7dee14364389871900670a02906a P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 13:56:57.275626 15738 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 13:56:57.276705 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:56:57.283737 13164 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 13:56:57.286527 25254 raft_consensus.cc:482] T db5568722b6344d18d6d6d271f05fc7a P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e47105a0f46e4511b7112e571688a4c7)
I20240503 13:56:57.286648 25254 raft_consensus.cc:504] T db5568722b6344d18d6d6d271f05fc7a P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:57.287035 25254 leader_election.cc:288] T db5568722b6344d18d6d6d271f05fc7a P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:57.288416 15492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5568722b6344d18d6d6d271f05fc7a" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546" is_pre_election: true
W20240503 13:56:57.292119 13249 leader_election.cc:334] T db5568722b6344d18d6d6d271f05fc7a P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:56:57.292201 13249 leader_election.cc:302] T db5568722b6344d18d6d6d271f05fc7a P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba0194098aa84449ae17e8a0d4c43703; no voters: 61b887f0d40d497badeca5604c1f0546, e47105a0f46e4511b7112e571688a4c7
I20240503 13:56:57.292663 25254 raft_consensus.cc:2726] T db5568722b6344d18d6d6d271f05fc7a P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 13:56:57.293309 13427 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 13:56:57.308806 12904 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:56:57.314193 25209 tablet.cc:1722] T 2427783ee0e348b98453818704ece5f8 P ba0194098aa84449ae17e8a0d4c43703: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:57.315465 25210 tablet.cc:1722] T 2427783ee0e348b98453818704ece5f8 P 61b887f0d40d497badeca5604c1f0546: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:57.322706 12907 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table_new_name [id=2ad9a0e46b12442cad28818be33337b2] alter complete (version 3)
I20240503 13:56:57.341977 23473 raft_consensus.cc:482] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e47105a0f46e4511b7112e571688a4c7)
I20240503 13:56:57.342109 23473 raft_consensus.cc:504] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:57.342483 23473 leader_election.cc:288] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:57.342959 15624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bdb9c14caf04abca39e56f8bcf1d1d4" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d" is_pre_election: true
W20240503 13:56:57.343919 13249 leader_election.cc:334] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:56:57.343993 13249 leader_election.cc:302] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba0194098aa84449ae17e8a0d4c43703; no voters: 009f849ee7b54c80a3f5264695047e4d, e47105a0f46e4511b7112e571688a4c7
I20240503 13:56:57.344125 23473 raft_consensus.cc:2726] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 13:56:57.377159 24637 raft_consensus.cc:482] T 0f9224c2e3b946d993a5fa5c95a1d681 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e47105a0f46e4511b7112e571688a4c7)
I20240503 13:56:57.377269 24637 raft_consensus.cc:504] T 0f9224c2e3b946d993a5fa5c95a1d681 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:57.377689 24637 leader_election.cc:288] T 0f9224c2e3b946d993a5fa5c95a1d681 P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:57.377959 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f9224c2e3b946d993a5fa5c95a1d681" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2" is_pre_election: true
I20240503 13:56:57.378629 24637 raft_consensus.cc:482] T 7dca80c8a74a42c4b459315878ef5d14 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e47105a0f46e4511b7112e571688a4c7)
I20240503 13:56:57.378711 24637 raft_consensus.cc:504] T 7dca80c8a74a42c4b459315878ef5d14 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
W20240503 13:56:57.378789 13380 leader_election.cc:334] T 0f9224c2e3b946d993a5fa5c95a1d681 P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:56:57.378837 13380 leader_election.cc:302] T 0f9224c2e3b946d993a5fa5c95a1d681 P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8; no voters: 76c386bef61a4bd5a907266a5118ffe2, e47105a0f46e4511b7112e571688a4c7
I20240503 13:56:57.378978 25056 raft_consensus.cc:2726] T 0f9224c2e3b946d993a5fa5c95a1d681 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 13:56:57.379014 24637 leader_election.cc:288] T 7dca80c8a74a42c4b459315878ef5d14 P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695)
I20240503 13:56:57.379333 13052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dca80c8a74a42c4b459315878ef5d14" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e" is_pre_election: true
W20240503 13:56:57.379917 13380 leader_election.cc:334] T 7dca80c8a74a42c4b459315878ef5d14 P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:56:57.380097 13380 leader_election.cc:302] T 7dca80c8a74a42c4b459315878ef5d14 P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8; no voters: 60ba3ffe3b7746469834cd91c84be14e, e47105a0f46e4511b7112e571688a4c7
I20240503 13:56:57.380229 25056 raft_consensus.cc:2726] T 7dca80c8a74a42c4b459315878ef5d14 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 13:56:57.381583 24959 raft_consensus.cc:482] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e47105a0f46e4511b7112e571688a4c7)
I20240503 13:56:57.381678 24959 raft_consensus.cc:504] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:57.381981 24959 leader_election.cc:288] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:57.382473 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f9224c2e3b946d993a5fa5c95a1d681" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
I20240503 13:56:57.382586 13445 raft_consensus.cc:2454] T 0f9224c2e3b946d993a5fa5c95a1d681 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 1.
W20240503 13:56:57.382851 15691 leader_election.cc:334] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:56:57.382987 15691 leader_election.cc:302] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, 76c386bef61a4bd5a907266a5118ffe2; no voters: e47105a0f46e4511b7112e571688a4c7
I20240503 13:56:57.383191 24959 raft_consensus.cc:2781] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 13:56:57.383239 24959 raft_consensus.cc:482] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e47105a0f46e4511b7112e571688a4c7)
I20240503 13:56:57.383275 24959 raft_consensus.cc:3037] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:56:57.384364 24959 raft_consensus.cc:504] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:57.384692 24959 leader_election.cc:288] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 2 election: Requested vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:57.385159 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f9224c2e3b946d993a5fa5c95a1d681" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
I20240503 13:56:57.385282 13445 raft_consensus.cc:3037] T 0f9224c2e3b946d993a5fa5c95a1d681 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Advancing to term 2
W20240503 13:56:57.385608 15691 leader_election.cc:334] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:56:57.386262 13445 raft_consensus.cc:2454] T 0f9224c2e3b946d993a5fa5c95a1d681 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 2.
I20240503 13:56:57.386556 15691 leader_election.cc:302] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, 76c386bef61a4bd5a907266a5118ffe2; no voters: e47105a0f46e4511b7112e571688a4c7
I20240503 13:56:57.386758 24959 raft_consensus.cc:2781] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:56:57.386839 24959 raft_consensus.cc:686] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 [term 2 LEADER]: Becoming Leader. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Running, Role: LEADER
I20240503 13:56:57.386974 24959 consensus_queue.cc:237] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:57.389421 12904 catalog_manager.cc:5484] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 reported cstate change: term changed from 1 to 2, leader changed from e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) to 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71). New cstate: current_term: 2 leader_uuid: "76c386bef61a4bd5a907266a5118ffe2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } }
I20240503 13:56:57.406065 25096 raft_consensus.cc:482] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e47105a0f46e4511b7112e571688a4c7)
I20240503 13:56:57.406193 25096 raft_consensus.cc:504] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:57.406558 25096 leader_election.cc:288] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:57.406864 13316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5568722b6344d18d6d6d271f05fc7a" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703" is_pre_election: true
I20240503 13:56:57.407032 13316 raft_consensus.cc:2454] T db5568722b6344d18d6d6d271f05fc7a P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 1.
I20240503 13:56:57.407284 24901 raft_consensus.cc:482] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e47105a0f46e4511b7112e571688a4c7)
I20240503 13:56:57.407393 24901 raft_consensus.cc:504] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:57.407474 15429 leader_election.cc:302] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61b887f0d40d497badeca5604c1f0546, ba0194098aa84449ae17e8a0d4c43703; no voters:
W20240503 13:56:57.407778 15426 leader_election.cc:334] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:56:57.407814 20722 raft_consensus.cc:2781] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 13:56:57.407866 20722 raft_consensus.cc:482] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e47105a0f46e4511b7112e571688a4c7)
I20240503 13:56:57.407900 20722 raft_consensus.cc:3037] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:56:57.408066 24901 leader_election.cc:288] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:57.408340 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10ee7dee14364389871900670a02906a" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2" is_pre_election: true
I20240503 13:56:57.408479 15757 raft_consensus.cc:2454] T 10ee7dee14364389871900670a02906a P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 1.
I20240503 13:56:57.408810 15561 leader_election.cc:302] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 76c386bef61a4bd5a907266a5118ffe2; no voters:
I20240503 13:56:57.408957 24901 raft_consensus.cc:2781] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 13:56:57.409006 24901 raft_consensus.cc:482] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Starting leader election (detected failure of leader e47105a0f46e4511b7112e571688a4c7)
I20240503 13:56:57.409060 24901 raft_consensus.cc:3037] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:56:57.409375 20722 raft_consensus.cc:504] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:57.409718 20722 leader_election.cc:288] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 2 election: Requested vote from peers ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:57.410109 24901 raft_consensus.cc:504] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
W20240503 13:56:57.410207 15558 leader_election.cc:334] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:56:57.410418 13316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db5568722b6344d18d6d6d271f05fc7a" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703"
I20240503 13:56:57.410535 13316 raft_consensus.cc:3037] T db5568722b6344d18d6d6d271f05fc7a P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:56:57.410691 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10ee7dee14364389871900670a02906a" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2"
I20240503 13:56:57.410802 15757 raft_consensus.cc:3037] T 10ee7dee14364389871900670a02906a P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Advancing to term 2
W20240503 13:56:57.410849 15426 leader_election.cc:334] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:56:57.411521 13316 raft_consensus.cc:2454] T db5568722b6344d18d6d6d271f05fc7a P ba0194098aa84449ae17e8a0d4c43703 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 2.
I20240503 13:56:57.412063 24901 leader_election.cc:288] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 2 election: Requested vote from peers 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
W20240503 13:56:57.412420 15558 leader_election.cc:334] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:56:57.412822 15429 leader_election.cc:302] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61b887f0d40d497badeca5604c1f0546, ba0194098aa84449ae17e8a0d4c43703; no voters: e47105a0f46e4511b7112e571688a4c7
I20240503 13:56:57.412994 20722 raft_consensus.cc:2781] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:56:57.413074 20722 raft_consensus.cc:686] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 [term 2 LEADER]: Becoming Leader. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Running, Role: LEADER
I20240503 13:56:57.413102 15757 raft_consensus.cc:2454] T 10ee7dee14364389871900670a02906a P 76c386bef61a4bd5a907266a5118ffe2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 2.
I20240503 13:56:57.413208 20722 consensus_queue.cc:237] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:57.416504 12904 catalog_manager.cc:5484] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 reported cstate change: term changed from 1 to 2, leader changed from e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) to 61b887f0d40d497badeca5604c1f0546 (127.6.98.69). New cstate: current_term: 2 leader_uuid: "61b887f0d40d497badeca5604c1f0546" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:57.416785 15561 leader_election.cc:302] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 76c386bef61a4bd5a907266a5118ffe2; no voters: e47105a0f46e4511b7112e571688a4c7
I20240503 13:56:57.416939 23621 raft_consensus.cc:2781] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:56:57.417025 23621 raft_consensus.cc:686] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d [term 2 LEADER]: Becoming Leader. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Running, Role: LEADER
I20240503 13:56:57.417153 23621 consensus_queue.cc:237] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:57.419394 12907 catalog_manager.cc:5484] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d reported cstate change: term changed from 1 to 2, leader changed from e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) to 009f849ee7b54c80a3f5264695047e4d (127.6.98.70). New cstate: current_term: 2 leader_uuid: "009f849ee7b54c80a3f5264695047e4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:57.426373 12907 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:57.426489 12907 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:57.428632 12907 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Sending DeleteTablet for 3 replicas of tablet 2427783ee0e348b98453818704ece5f8
I20240503 13:56:57.429211 13295 tablet_service.cc:1514] Processing DeleteTablet for tablet 2427783ee0e348b98453818704ece5f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:57 UTC) from {username='slave'} at 127.0.0.1:33250
I20240503 13:56:57.429318 15473 tablet_service.cc:1514] Processing DeleteTablet for tablet 2427783ee0e348b98453818704ece5f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:57 UTC) from {username='slave'} at 127.0.0.1:58758
I20240503 13:56:57.429626 25272 tablet_replica.cc:330] T 2427783ee0e348b98453818704ece5f8 P 61b887f0d40d497badeca5604c1f0546: stopping tablet replica
I20240503 13:56:57.429803 25272 raft_consensus.cc:2229] T 2427783ee0e348b98453818704ece5f8 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:57.429872 25271 tablet_replica.cc:330] T 2427783ee0e348b98453818704ece5f8 P ba0194098aa84449ae17e8a0d4c43703: stopping tablet replica
I20240503 13:56:57.429956 25272 raft_consensus.cc:2258] T 2427783ee0e348b98453818704ece5f8 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:57.430047 25271 raft_consensus.cc:2229] T 2427783ee0e348b98453818704ece5f8 P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:56:57.430301 25271 raft_consensus.cc:2258] T 2427783ee0e348b98453818704ece5f8 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:57.430608 25272 ts_tablet_manager.cc:1875] T 2427783ee0e348b98453818704ece5f8 P 61b887f0d40d497badeca5604c1f0546: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:57.430934 25271 ts_tablet_manager.cc:1875] T 2427783ee0e348b98453818704ece5f8 P ba0194098aa84449ae17e8a0d4c43703: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:57.432590 25272 ts_tablet_manager.cc:1888] T 2427783ee0e348b98453818704ece5f8 P 61b887f0d40d497badeca5604c1f0546: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:57.432654 25272 log.cc:1196] T 2427783ee0e348b98453818704ece5f8 P 61b887f0d40d497badeca5604c1f0546: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/wal/wals/2427783ee0e348b98453818704ece5f8
I20240503 13:56:57.432849 25272 ts_tablet_manager.cc:1909] T 2427783ee0e348b98453818704ece5f8 P 61b887f0d40d497badeca5604c1f0546: Deleting consensus metadata
I20240503 13:56:57.432849 25271 ts_tablet_manager.cc:1888] T 2427783ee0e348b98453818704ece5f8 P ba0194098aa84449ae17e8a0d4c43703: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:57.432897 25271 log.cc:1196] T 2427783ee0e348b98453818704ece5f8 P ba0194098aa84449ae17e8a0d4c43703: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-2/wal/wals/2427783ee0e348b98453818704ece5f8
I20240503 13:56:57.433167 24959 raft_consensus.cc:482] T eb92a057165f4da7b96d910d847ff761 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e47105a0f46e4511b7112e571688a4c7)
I20240503 13:56:57.433324 25271 ts_tablet_manager.cc:1909] T 2427783ee0e348b98453818704ece5f8 P ba0194098aa84449ae17e8a0d4c43703: Deleting consensus metadata
I20240503 13:56:57.433287 24959 raft_consensus.cc:504] T eb92a057165f4da7b96d910d847ff761 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:57.433748 12899 catalog_manager.cc:4830] TS 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153): tablet 2427783ee0e348b98453818704ece5f8 (table rebalancer_dynamic_table_new_name [id=2ad9a0e46b12442cad28818be33337b2]) successfully deleted
I20240503 13:56:57.434012 12899 catalog_manager.cc:4830] TS ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279): tablet 2427783ee0e348b98453818704ece5f8 (table rebalancer_dynamic_table_new_name [id=2ad9a0e46b12442cad28818be33337b2]) successfully deleted
I20240503 13:56:57.434299 24959 leader_election.cc:288] T eb92a057165f4da7b96d910d847ff761 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:57.434727 15492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb92a057165f4da7b96d910d847ff761" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546" is_pre_election: true
W20240503 13:56:57.435796 15691 leader_election.cc:334] T eb92a057165f4da7b96d910d847ff761 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:56:57.435875 15691 leader_election.cc:302] T eb92a057165f4da7b96d910d847ff761 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76c386bef61a4bd5a907266a5118ffe2; no voters: 61b887f0d40d497badeca5604c1f0546, e47105a0f46e4511b7112e571688a4c7
I20240503 13:56:57.436126 24959 raft_consensus.cc:2726] T eb92a057165f4da7b96d910d847ff761 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 13:56:57.439402 24809 raft_consensus.cc:482] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e47105a0f46e4511b7112e571688a4c7)
I20240503 13:56:57.439523 24809 raft_consensus.cc:504] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:57.439868 24809 leader_election.cc:288] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:57.440295 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f58fc3bc579748bb88de34612808ccda" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
I20240503 13:56:57.440450 13445 raft_consensus.cc:2454] T f58fc3bc579748bb88de34612808ccda P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 1.
I20240503 13:56:57.440899 12987 leader_election.cc:302] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 7690ee79577c45e3a2c8f8b552c0bff8; no voters:
W20240503 13:56:57.441028 12987 leader_election.cc:334] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:56:57.441035 24809 raft_consensus.cc:2781] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 13:56:57.441100 24809 raft_consensus.cc:482] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Starting leader election (detected failure of leader e47105a0f46e4511b7112e571688a4c7)
I20240503 13:56:57.441148 24809 raft_consensus.cc:3037] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:56:57.441951 24809 raft_consensus.cc:504] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:57.442307 24809 leader_election.cc:288] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 2 election: Requested vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:57.442741 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f58fc3bc579748bb88de34612808ccda" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
I20240503 13:56:57.442867 13445 raft_consensus.cc:3037] T f58fc3bc579748bb88de34612808ccda P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Advancing to term 2
W20240503 13:56:57.443192 12987 leader_election.cc:334] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:56:57.443856 13445 raft_consensus.cc:2454] T f58fc3bc579748bb88de34612808ccda P 7690ee79577c45e3a2c8f8b552c0bff8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 2.
I20240503 13:56:57.444118 12987 leader_election.cc:302] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 7690ee79577c45e3a2c8f8b552c0bff8; no voters: e47105a0f46e4511b7112e571688a4c7
I20240503 13:56:57.444265 25002 raft_consensus.cc:2781] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:56:57.444343 25002 raft_consensus.cc:686] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e [term 2 LEADER]: Becoming Leader. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Running, Role: LEADER
I20240503 13:56:57.444478 25002 consensus_queue.cc:237] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:57.447173 12907 catalog_manager.cc:5484] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e reported cstate change: term changed from 1 to 2, leader changed from e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) to 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65). New cstate: current_term: 2 leader_uuid: "60ba3ffe3b7746469834cd91c84be14e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:57.447453 25269 raft_consensus.cc:482] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e47105a0f46e4511b7112e571688a4c7)
I20240503 13:56:57.447579 25269 raft_consensus.cc:504] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:57.447960 25269 leader_election.cc:288] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:57.448746 13316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "557f14aaf5564ebaa461720a4d7997ff" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703" is_pre_election: true
I20240503 13:56:57.448892 13316 raft_consensus.cc:2454] T 557f14aaf5564ebaa461720a4d7997ff P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 1.
W20240503 13:56:57.449100 15558 leader_election.cc:334] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:56:57.449200 15561 leader_election.cc:302] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, ba0194098aa84449ae17e8a0d4c43703; no voters: e47105a0f46e4511b7112e571688a4c7
I20240503 13:56:57.449436 25269 raft_consensus.cc:2781] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 13:56:57.449492 25269 raft_consensus.cc:482] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Starting leader election (detected failure of leader e47105a0f46e4511b7112e571688a4c7)
I20240503 13:56:57.449542 25269 raft_consensus.cc:3037] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:56:57.450528 25269 raft_consensus.cc:504] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:57.450667 23473 raft_consensus.cc:482] T 3a140864f59b483bb5fea3de83d28ecf P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e47105a0f46e4511b7112e571688a4c7)
I20240503 13:56:57.450762 23473 raft_consensus.cc:504] T 3a140864f59b483bb5fea3de83d28ecf P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:57.451150 23473 leader_election.cc:288] T 3a140864f59b483bb5fea3de83d28ecf P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:57.451187 25269 leader_election.cc:288] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 2 election: Requested vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:57.451536 15624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a140864f59b483bb5fea3de83d28ecf" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d" is_pre_election: true
I20240503 13:56:57.451682 13316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "557f14aaf5564ebaa461720a4d7997ff" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703"
I20240503 13:56:57.451781 13316 raft_consensus.cc:3037] T 557f14aaf5564ebaa461720a4d7997ff P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Advancing to term 2
W20240503 13:56:57.452726 13249 leader_election.cc:334] T 3a140864f59b483bb5fea3de83d28ecf P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:56:57.452787 13249 leader_election.cc:302] T 3a140864f59b483bb5fea3de83d28ecf P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba0194098aa84449ae17e8a0d4c43703; no voters: 009f849ee7b54c80a3f5264695047e4d, e47105a0f46e4511b7112e571688a4c7
I20240503 13:56:57.452885 13316 raft_consensus.cc:2454] T 557f14aaf5564ebaa461720a4d7997ff P ba0194098aa84449ae17e8a0d4c43703 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 2.
I20240503 13:56:57.453058 23473 raft_consensus.cc:2726] T 3a140864f59b483bb5fea3de83d28ecf P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 13:56:57.453155 15561 leader_election.cc:302] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, ba0194098aa84449ae17e8a0d4c43703; no voters:
W20240503 13:56:57.453277 15558 leader_election.cc:334] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:56:57.453786 25269 raft_consensus.cc:2781] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:56:57.453882 25269 raft_consensus.cc:686] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d [term 2 LEADER]: Becoming Leader. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Running, Role: LEADER
I20240503 13:56:57.454059 25269 consensus_queue.cc:237] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:57.456240 12907 catalog_manager.cc:5484] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d reported cstate change: term changed from 1 to 2, leader changed from e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) to 009f849ee7b54c80a3f5264695047e4d (127.6.98.70). New cstate: current_term: 2 leader_uuid: "009f849ee7b54c80a3f5264695047e4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:57.474041 20722 raft_consensus.cc:482] T 5f34ac73ebe544a1b7a34e8c2d11568e P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e47105a0f46e4511b7112e571688a4c7)
I20240503 13:56:57.474162 20722 raft_consensus.cc:504] T 5f34ac73ebe544a1b7a34e8c2d11568e P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:57.474601 20722 leader_election.cc:288] T 5f34ac73ebe544a1b7a34e8c2d11568e P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:57.474975 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f34ac73ebe544a1b7a34e8c2d11568e" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
W20240503 13:56:57.475746 15426 leader_election.cc:334] T 5f34ac73ebe544a1b7a34e8c2d11568e P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:56:57.475831 15426 leader_election.cc:302] T 5f34ac73ebe544a1b7a34e8c2d11568e P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61b887f0d40d497badeca5604c1f0546; no voters: 7690ee79577c45e3a2c8f8b552c0bff8, e47105a0f46e4511b7112e571688a4c7
I20240503 13:56:57.475962 20722 raft_consensus.cc:2726] T 5f34ac73ebe544a1b7a34e8c2d11568e P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 13:56:57.480384 25269 raft_consensus.cc:482] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e47105a0f46e4511b7112e571688a4c7)
I20240503 13:56:57.480533 25269 raft_consensus.cc:504] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:57.480988 25269 leader_election.cc:288] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:57.481734 13316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a140864f59b483bb5fea3de83d28ecf" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703" is_pre_election: true
I20240503 13:56:57.481894 13316 raft_consensus.cc:2454] T 3a140864f59b483bb5fea3de83d28ecf P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 1.
W20240503 13:56:57.482152 15558 leader_election.cc:334] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:56:57.482247 15561 leader_election.cc:302] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, ba0194098aa84449ae17e8a0d4c43703; no voters: e47105a0f46e4511b7112e571688a4c7
I20240503 13:56:57.482447 25269 raft_consensus.cc:2781] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 13:56:57.482525 25269 raft_consensus.cc:482] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Starting leader election (detected failure of leader e47105a0f46e4511b7112e571688a4c7)
I20240503 13:56:57.482573 25269 raft_consensus.cc:3037] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:56:57.483664 25269 raft_consensus.cc:504] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:57.484014 25269 leader_election.cc:288] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 2 election: Requested vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:57.484333 13316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a140864f59b483bb5fea3de83d28ecf" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703"
I20240503 13:56:57.484449 13316 raft_consensus.cc:3037] T 3a140864f59b483bb5fea3de83d28ecf P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Advancing to term 2
W20240503 13:56:57.484920 15558 leader_election.cc:334] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:56:57.485368 13316 raft_consensus.cc:2454] T 3a140864f59b483bb5fea3de83d28ecf P ba0194098aa84449ae17e8a0d4c43703 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 2.
I20240503 13:56:57.485672 15561 leader_election.cc:302] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, ba0194098aa84449ae17e8a0d4c43703; no voters: e47105a0f46e4511b7112e571688a4c7
I20240503 13:56:57.485832 25269 raft_consensus.cc:2781] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:56:57.485924 25269 raft_consensus.cc:686] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d [term 2 LEADER]: Becoming Leader. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Running, Role: LEADER
I20240503 13:56:57.486085 25269 consensus_queue.cc:237] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:57.488332 12907 catalog_manager.cc:5484] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d reported cstate change: term changed from 1 to 2, leader changed from e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) to 009f849ee7b54c80a3f5264695047e4d (127.6.98.70). New cstate: current_term: 2 leader_uuid: "009f849ee7b54c80a3f5264695047e4d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:57.490906 24959 raft_consensus.cc:482] T 4b4b26adad184e1ea57c00a44635679c P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e47105a0f46e4511b7112e571688a4c7)
I20240503 13:56:57.491003 24959 raft_consensus.cc:504] T 4b4b26adad184e1ea57c00a44635679c P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:57.491310 24959 leader_election.cc:288] T 4b4b26adad184e1ea57c00a44635679c P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:57.491914 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b4b26adad184e1ea57c00a44635679c" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
W20240503 13:56:57.492305 15691 leader_election.cc:334] T 4b4b26adad184e1ea57c00a44635679c P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:56:57.492365 15691 leader_election.cc:302] T 4b4b26adad184e1ea57c00a44635679c P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76c386bef61a4bd5a907266a5118ffe2; no voters: 7690ee79577c45e3a2c8f8b552c0bff8, e47105a0f46e4511b7112e571688a4c7
I20240503 13:56:57.492516 24959 raft_consensus.cc:2726] T 4b4b26adad184e1ea57c00a44635679c P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 13:56:57.494381 25096 raft_consensus.cc:482] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e47105a0f46e4511b7112e571688a4c7)
I20240503 13:56:57.494490 25096 raft_consensus.cc:504] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:57.494858 25096 leader_election.cc:288] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885)
I20240503 13:56:57.495347 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb92a057165f4da7b96d910d847ff761" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2" is_pre_election: true
I20240503 13:56:57.495548 15757 raft_consensus.cc:2454] T eb92a057165f4da7b96d910d847ff761 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 1.
I20240503 13:56:57.496145 15429 leader_election.cc:302] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61b887f0d40d497badeca5604c1f0546, 76c386bef61a4bd5a907266a5118ffe2; no voters:
W20240503 13:56:57.496296 15426 leader_election.cc:334] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:56:57.496340 25096 raft_consensus.cc:2781] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 13:56:57.496385 25096 raft_consensus.cc:482] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e47105a0f46e4511b7112e571688a4c7)
I20240503 13:56:57.496419 25096 raft_consensus.cc:3037] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:56:57.497400 25096 raft_consensus.cc:504] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:57.497679 25096 leader_election.cc:288] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 2 election: Requested vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885)
I20240503 13:56:57.498441 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb92a057165f4da7b96d910d847ff761" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2"
I20240503 13:56:57.498569 15757 raft_consensus.cc:3037] T eb92a057165f4da7b96d910d847ff761 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Advancing to term 2
W20240503 13:56:57.498725 15426 leader_election.cc:334] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:56:57.499382 15757 raft_consensus.cc:2454] T eb92a057165f4da7b96d910d847ff761 P 76c386bef61a4bd5a907266a5118ffe2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 2.
I20240503 13:56:57.499663 15429 leader_election.cc:302] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61b887f0d40d497badeca5604c1f0546, 76c386bef61a4bd5a907266a5118ffe2; no voters: e47105a0f46e4511b7112e571688a4c7
I20240503 13:56:57.499836 25096 raft_consensus.cc:2781] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:56:57.499918 25096 raft_consensus.cc:686] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 [term 2 LEADER]: Becoming Leader. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Running, Role: LEADER
I20240503 13:56:57.500065 25096 consensus_queue.cc:237] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:57.502491 12907 catalog_manager.cc:5484] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 reported cstate change: term changed from 1 to 2, leader changed from e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) to 61b887f0d40d497badeca5604c1f0546 (127.6.98.69). New cstate: current_term: 2 leader_uuid: "61b887f0d40d497badeca5604c1f0546" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:57.523782 24831 consensus_queue.cc:1035] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:57.526664 25002 raft_consensus.cc:482] T dd39f3ab7cdb42b38267214056605328 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e47105a0f46e4511b7112e571688a4c7)
I20240503 13:56:57.526803 25002 raft_consensus.cc:504] T dd39f3ab7cdb42b38267214056605328 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
W20240503 13:56:57.527070 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:57.527196 25002 leader_election.cc:288] T dd39f3ab7cdb42b38267214056605328 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951)
I20240503 13:56:57.527906 15624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd39f3ab7cdb42b38267214056605328" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d" is_pre_election: true
W20240503 13:56:57.528735 12987 leader_election.cc:334] T dd39f3ab7cdb42b38267214056605328 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:56:57.528800 12987 leader_election.cc:302] T dd39f3ab7cdb42b38267214056605328 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e; no voters: 009f849ee7b54c80a3f5264695047e4d, e47105a0f46e4511b7112e571688a4c7
I20240503 13:56:57.529109 25002 raft_consensus.cc:2726] T dd39f3ab7cdb42b38267214056605328 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 13:56:57.530184 12904 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:56:57.531141 25056 raft_consensus.cc:482] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e47105a0f46e4511b7112e571688a4c7)
I20240503 13:56:57.531253 25056 raft_consensus.cc:504] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:57.531621 25056 leader_election.cc:288] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885)
I20240503 13:56:57.532265 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b4b26adad184e1ea57c00a44635679c" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2" is_pre_election: true
I20240503 13:56:57.532408 15757 raft_consensus.cc:2454] T 4b4b26adad184e1ea57c00a44635679c P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 1.
W20240503 13:56:57.532917 13380 leader_election.cc:334] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:56:57.533953 13383 leader_election.cc:302] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, 76c386bef61a4bd5a907266a5118ffe2; no voters: e47105a0f46e4511b7112e571688a4c7
I20240503 13:56:57.534164 25056 raft_consensus.cc:2781] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 13:56:57.534207 25056 raft_consensus.cc:482] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e47105a0f46e4511b7112e571688a4c7)
I20240503 13:56:57.534245 25056 raft_consensus.cc:3037] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:56:57.535380 25056 raft_consensus.cc:504] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:57.535724 25056 leader_election.cc:288] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 2 election: Requested vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885)
I20240503 13:56:57.536058 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b4b26adad184e1ea57c00a44635679c" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2"
I20240503 13:56:57.536170 15757 raft_consensus.cc:3037] T 4b4b26adad184e1ea57c00a44635679c P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:56:57.536808 15473 tablet_service.cc:1467] Processing CreateTablet for tablet a21b33de70e94930bea00c0e16e713fc (DEFAULT_TABLE table=rebalancer_dynamic_table [id=41dc105ceda04fabbfa4246ecf873aed]), partition=
I20240503 13:56:57.536954 13427 tablet_service.cc:1467] Processing CreateTablet for tablet a21b33de70e94930bea00c0e16e713fc (DEFAULT_TABLE table=rebalancer_dynamic_table [id=41dc105ceda04fabbfa4246ecf873aed]), partition=
I20240503 13:56:57.537063 15473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a21b33de70e94930bea00c0e16e713fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:57.537179 13427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a21b33de70e94930bea00c0e16e713fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:57.537228 15757 raft_consensus.cc:2454] T 4b4b26adad184e1ea57c00a44635679c P 76c386bef61a4bd5a907266a5118ffe2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 2.
I20240503 13:56:57.537603 13383 leader_election.cc:302] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7690ee79577c45e3a2c8f8b552c0bff8, 76c386bef61a4bd5a907266a5118ffe2; no voters:
W20240503 13:56:57.538164 12896 catalog_manager.cc:4573] TS e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): CreateTablet RPC failed for tablet a21b33de70e94930bea00c0e16e713fc: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:56:57.539465 25056 raft_consensus.cc:2781] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 [term 2 FOLLOWER]: Leader election won for term 2
W20240503 13:56:57.539552 13380 leader_election.cc:334] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:56:57.539646 25056 raft_consensus.cc:686] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 [term 2 LEADER]: Becoming Leader. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Running, Role: LEADER
I20240503 13:56:57.539781 25056 consensus_queue.cc:237] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } }
I20240503 13:56:57.539975 25281 tablet_bootstrap.cc:492] T a21b33de70e94930bea00c0e16e713fc P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:57.539929 23473 consensus_queue.cc:1035] T 67395f0e6df94bd49aa76359858ad327 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:57.541424 25280 tablet_bootstrap.cc:492] T a21b33de70e94930bea00c0e16e713fc P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:57.542793 12904 catalog_manager.cc:5484] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 reported cstate change: term changed from 1 to 2, leader changed from e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) to 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68). New cstate: current_term: 2 leader_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } }
I20240503 13:56:57.543453 25269 raft_consensus.cc:482] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e47105a0f46e4511b7112e571688a4c7)
I20240503 13:56:57.543555 25269 raft_consensus.cc:504] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:57.543066 25280 tablet_bootstrap.cc:654] T a21b33de70e94930bea00c0e16e713fc P 61b887f0d40d497badeca5604c1f0546: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:57.544009 25269 leader_election.cc:288] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:57.545092 13051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd39f3ab7cdb42b38267214056605328" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e" is_pre_election: true
I20240503 13:56:57.545236 13051 raft_consensus.cc:2454] T dd39f3ab7cdb42b38267214056605328 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 1.
I20240503 13:56:57.545542 15559 leader_election.cc:302] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 60ba3ffe3b7746469834cd91c84be14e; no voters:
I20240503 13:56:57.545887 23621 raft_consensus.cc:2781] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 13:56:57.545957 23621 raft_consensus.cc:482] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Starting leader election (detected failure of leader e47105a0f46e4511b7112e571688a4c7)
I20240503 13:56:57.545998 23621 raft_consensus.cc:3037] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:56:57.546681 25281 tablet_bootstrap.cc:654] T a21b33de70e94930bea00c0e16e713fc P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:57.547070 23621 raft_consensus.cc:504] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:57.547447 23621 leader_election.cc:288] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 2 election: Requested vote from peers 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65:34695), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:57.547914 25280 tablet_bootstrap.cc:492] T a21b33de70e94930bea00c0e16e713fc P 61b887f0d40d497badeca5604c1f0546: No bootstrap required, opened a new log
I20240503 13:56:57.548012 25280 ts_tablet_manager.cc:1367] T a21b33de70e94930bea00c0e16e713fc P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
W20240503 13:56:57.548306 15558 leader_election.cc:334] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:56:57.549515 25281 tablet_bootstrap.cc:492] T a21b33de70e94930bea00c0e16e713fc P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:57.549611 25281 ts_tablet_manager.cc:1367] T a21b33de70e94930bea00c0e16e713fc P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
W20240503 13:56:57.549921 15558 leader_election.cc:334] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:56:57.550081 25281 raft_consensus.cc:348] T a21b33de70e94930bea00c0e16e713fc P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:57.550163 25281 raft_consensus.cc:374] T a21b33de70e94930bea00c0e16e713fc P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:57.550181 25281 raft_consensus.cc:729] T a21b33de70e94930bea00c0e16e713fc P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:57.550295 25281 consensus_queue.cc:260] T a21b33de70e94930bea00c0e16e713fc P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:57.550537 25281 ts_tablet_manager.cc:1398] T a21b33de70e94930bea00c0e16e713fc P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:57.550683 25056 raft_consensus.cc:482] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e47105a0f46e4511b7112e571688a4c7)
I20240503 13:56:57.550753 25056 raft_consensus.cc:504] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:57.551075 25056 leader_election.cc:288] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153)
I20240503 13:56:57.551167 25280 raft_consensus.cc:348] T a21b33de70e94930bea00c0e16e713fc P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:57.551270 25280 raft_consensus.cc:374] T a21b33de70e94930bea00c0e16e713fc P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:57.551301 25280 raft_consensus.cc:729] T a21b33de70e94930bea00c0e16e713fc P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:57.551440 25280 consensus_queue.cc:260] T a21b33de70e94930bea00c0e16e713fc P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:57.551666 15492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f34ac73ebe544a1b7a34e8c2d11568e" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546" is_pre_election: true
I20240503 13:56:57.551769 15492 raft_consensus.cc:2454] T 5f34ac73ebe544a1b7a34e8c2d11568e P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 1.
I20240503 13:56:57.552469 25096 raft_consensus.cc:482] T a6a94e83461643d4adfc4abc7dfb74e7 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e47105a0f46e4511b7112e571688a4c7)
I20240503 13:56:57.552583 25096 raft_consensus.cc:504] T a6a94e83461643d4adfc4abc7dfb74e7 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:57.552639 25280 ts_tablet_manager.cc:1398] T a21b33de70e94930bea00c0e16e713fc P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240503 13:56:57.552790 13383 leader_election.cc:302] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61b887f0d40d497badeca5604c1f0546, 7690ee79577c45e3a2c8f8b552c0bff8; no voters:
I20240503 13:56:57.552920 25096 leader_election.cc:288] T a6a94e83461643d4adfc4abc7dfb74e7 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:57.553056 25056 raft_consensus.cc:2781] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 13:56:57.553100 25056 raft_consensus.cc:482] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e47105a0f46e4511b7112e571688a4c7)
I20240503 13:56:57.553131 25056 raft_consensus.cc:3037] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:56:57.553463 13182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a94e83461643d4adfc4abc7dfb74e7" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770" is_pre_election: true
I20240503 13:56:57.554195 25056 raft_consensus.cc:504] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:57.554525 25056 leader_election.cc:288] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 2 election: Requested vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153)
I20240503 13:56:57.554872 15492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f34ac73ebe544a1b7a34e8c2d11568e" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546"
I20240503 13:56:57.554967 15492 raft_consensus.cc:3037] T 5f34ac73ebe544a1b7a34e8c2d11568e P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Advancing to term 2
W20240503 13:56:57.555008 13380 leader_election.cc:334] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:56:57.555752 13051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd39f3ab7cdb42b38267214056605328" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60ba3ffe3b7746469834cd91c84be14e"
I20240503 13:56:57.555900 13051 raft_consensus.cc:3037] T dd39f3ab7cdb42b38267214056605328 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:56:57.556020 15492 raft_consensus.cc:2454] T 5f34ac73ebe544a1b7a34e8c2d11568e P 61b887f0d40d497badeca5604c1f0546 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 2.
I20240503 13:56:57.557013 13051 raft_consensus.cc:2454] T dd39f3ab7cdb42b38267214056605328 P 60ba3ffe3b7746469834cd91c84be14e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 2.
I20240503 13:56:57.557440 24831 raft_consensus.cc:482] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e47105a0f46e4511b7112e571688a4c7)
I20240503 13:56:57.557543 24831 raft_consensus.cc:504] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
W20240503 13:56:57.557816 13380 leader_election.cc:334] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:56:57.557916 24831 leader_election.cc:288] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:57.558856 15624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3849e4effe64656bd8b0f504c9dc17a" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d" is_pre_election: true
I20240503 13:56:57.558995 15624 raft_consensus.cc:2454] T d3849e4effe64656bd8b0f504c9dc17a P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 1.
I20240503 13:56:57.559123 15559 leader_election.cc:302] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 60ba3ffe3b7746469834cd91c84be14e; no voters: e47105a0f46e4511b7112e571688a4c7
I20240503 13:56:57.559288 23621 raft_consensus.cc:2781] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:56:57.559383 23621 raft_consensus.cc:686] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d [term 2 LEADER]: Becoming Leader. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Running, Role: LEADER
I20240503 13:56:57.559525 23621 consensus_queue.cc:237] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } }
I20240503 13:56:57.559631 13118 leader_election.cc:302] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 8ccf5a4ac450414c9d3579bd5800b770; no voters:
I20240503 13:56:57.559651 23473 consensus_queue.cc:1035] T 67395f0e6df94bd49aa76359858ad327 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:57.559769 24831 raft_consensus.cc:2781] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 13:56:57.559827 24831 raft_consensus.cc:482] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e47105a0f46e4511b7112e571688a4c7)
I20240503 13:56:57.559862 24831 raft_consensus.cc:3037] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:56:57.560288 13383 leader_election.cc:302] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61b887f0d40d497badeca5604c1f0546, 7690ee79577c45e3a2c8f8b552c0bff8; no voters: e47105a0f46e4511b7112e571688a4c7
I20240503 13:56:57.560956 24831 raft_consensus.cc:504] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:57.561096 25056 raft_consensus.cc:2781] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:56:57.561168 25056 raft_consensus.cc:686] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 [term 2 LEADER]: Becoming Leader. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Running, Role: LEADER
I20240503 13:56:57.561264 24831 leader_election.cc:288] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 2 election: Requested vote from peers 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:57.561277 25056 consensus_queue.cc:237] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
W20240503 13:56:57.559769 13118 leader_election.cc:334] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:56:57.563496 15426 leader_election.cc:334] T a6a94e83461643d4adfc4abc7dfb74e7 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:56:57.563584 15426 leader_election.cc:302] T a6a94e83461643d4adfc4abc7dfb74e7 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61b887f0d40d497badeca5604c1f0546; no voters: 8ccf5a4ac450414c9d3579bd5800b770, e47105a0f46e4511b7112e571688a4c7
I20240503 13:56:57.563716 25096 raft_consensus.cc:2726] T a6a94e83461643d4adfc4abc7dfb74e7 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 13:56:57.563921 15624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3849e4effe64656bd8b0f504c9dc17a" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d"
I20240503 13:56:57.564036 15624 raft_consensus.cc:3037] T d3849e4effe64656bd8b0f504c9dc17a P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:56:57.565140 15624 raft_consensus.cc:2454] T d3849e4effe64656bd8b0f504c9dc17a P 009f849ee7b54c80a3f5264695047e4d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 2.
I20240503 13:56:57.566620 12906 catalog_manager.cc:5484] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 reported cstate change: term changed from 1 to 2, leader changed from e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) to 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68). New cstate: current_term: 2 leader_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:57.568445 12907 catalog_manager.cc:5484] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d reported cstate change: term changed from 1 to 2, leader changed from e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) to 009f849ee7b54c80a3f5264695047e4d (127.6.98.70). New cstate: current_term: 2 leader_uuid: "009f849ee7b54c80a3f5264695047e4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } }
I20240503 13:56:57.570081 13118 leader_election.cc:302] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 8ccf5a4ac450414c9d3579bd5800b770; no voters:
I20240503 13:56:57.570453 25016 raft_consensus.cc:2781] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:56:57.570549 25016 raft_consensus.cc:686] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 [term 2 LEADER]: Becoming Leader. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Running, Role: LEADER
I20240503 13:56:57.570672 25016 consensus_queue.cc:237] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
W20240503 13:56:57.571090 13118 leader_election.cc:334] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:56:57.574170 12906 catalog_manager.cc:5484] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 reported cstate change: term changed from 1 to 2, leader changed from e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) to 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66). New cstate: current_term: 2 leader_uuid: "8ccf5a4ac450414c9d3579bd5800b770" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } }
I20240503 13:56:57.575122 25016 raft_consensus.cc:482] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e47105a0f46e4511b7112e571688a4c7)
I20240503 13:56:57.575233 25016 raft_consensus.cc:504] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:57.575618 25016 leader_election.cc:288] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:57.576069 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1fed738e1c46a6a0a0f9e2b9cfd9b0" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2" is_pre_election: true
W20240503 13:56:57.580777 13118 leader_election.cc:334] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:56:57.580869 13118 leader_election.cc:302] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ccf5a4ac450414c9d3579bd5800b770; no voters: 76c386bef61a4bd5a907266a5118ffe2, e47105a0f46e4511b7112e571688a4c7
I20240503 13:56:57.581195 25016 raft_consensus.cc:2726] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 13:56:57.588829 25056 raft_consensus.cc:482] T a21b33de70e94930bea00c0e16e713fc P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:57.588946 25056 raft_consensus.cc:504] T a21b33de70e94930bea00c0e16e713fc P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:57.589330 25056 leader_election.cc:288] T a21b33de70e94930bea00c0e16e713fc P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153)
I20240503 13:56:57.590046 15492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21b33de70e94930bea00c0e16e713fc" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546" is_pre_election: true
I20240503 13:56:57.590211 15492 raft_consensus.cc:2454] T a21b33de70e94930bea00c0e16e713fc P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 0.
I20240503 13:56:57.590505 13383 leader_election.cc:302] T a21b33de70e94930bea00c0e16e713fc P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61b887f0d40d497badeca5604c1f0546, 7690ee79577c45e3a2c8f8b552c0bff8; no voters:
I20240503 13:56:57.591120 25056 raft_consensus.cc:2781] T a21b33de70e94930bea00c0e16e713fc P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:57.591178 25056 raft_consensus.cc:482] T a21b33de70e94930bea00c0e16e713fc P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:57.591213 25056 raft_consensus.cc:3037] T a21b33de70e94930bea00c0e16e713fc P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:57.592298 25056 raft_consensus.cc:504] T a21b33de70e94930bea00c0e16e713fc P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:57.592680 25056 leader_election.cc:288] T a21b33de70e94930bea00c0e16e713fc P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 election: Requested vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153)
I20240503 13:56:57.593005 15492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21b33de70e94930bea00c0e16e713fc" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546"
I20240503 13:56:57.593123 15492 raft_consensus.cc:3037] T a21b33de70e94930bea00c0e16e713fc P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Advancing to term 1
W20240503 13:56:57.593250 13380 leader_election.cc:334] T a21b33de70e94930bea00c0e16e713fc P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:56:57.594090 15492 raft_consensus.cc:2454] T a21b33de70e94930bea00c0e16e713fc P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 1.
I20240503 13:56:57.594420 13383 leader_election.cc:302] T a21b33de70e94930bea00c0e16e713fc P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61b887f0d40d497badeca5604c1f0546, 7690ee79577c45e3a2c8f8b552c0bff8; no voters:
I20240503 13:56:57.594583 25056 raft_consensus.cc:2781] T a21b33de70e94930bea00c0e16e713fc P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:57.594642 25056 raft_consensus.cc:686] T a21b33de70e94930bea00c0e16e713fc P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 LEADER]: Becoming Leader. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Running, Role: LEADER
I20240503 13:56:57.594761 25056 consensus_queue.cc:237] T a21b33de70e94930bea00c0e16e713fc P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
W20240503 13:56:57.595093 13380 leader_election.cc:334] T a21b33de70e94930bea00c0e16e713fc P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:56:57.599462 12906 catalog_manager.cc:5484] T a21b33de70e94930bea00c0e16e713fc P 7690ee79577c45e3a2c8f8b552c0bff8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68). New cstate: current_term: 1 leader_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:57.637286 25096 raft_consensus.cc:482] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e47105a0f46e4511b7112e571688a4c7)
I20240503 13:56:57.637414 25096 raft_consensus.cc:504] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:57.637814 25096 leader_election.cc:288] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885)
I20240503 13:56:57.638406 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25d41dbdacc341ee93c86f7a7f056123" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2" is_pre_election: true
I20240503 13:56:57.638545 15757 raft_consensus.cc:2454] T 25d41dbdacc341ee93c86f7a7f056123 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 1.
I20240503 13:56:57.638828 15429 leader_election.cc:302] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61b887f0d40d497badeca5604c1f0546, 76c386bef61a4bd5a907266a5118ffe2; no voters:
I20240503 13:56:57.639004 25096 raft_consensus.cc:2781] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 13:56:57.639052 25096 raft_consensus.cc:482] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e47105a0f46e4511b7112e571688a4c7)
I20240503 13:56:57.639094 25096 raft_consensus.cc:3037] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Advancing to term 2
W20240503 13:56:57.639173 15426 leader_election.cc:334] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:56:57.640241 25096 raft_consensus.cc:504] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:57.641364 25096 leader_election.cc:288] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 2 election: Requested vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885)
I20240503 13:56:57.641685 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25d41dbdacc341ee93c86f7a7f056123" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2"
I20240503 13:56:57.641817 15757 raft_consensus.cc:3037] T 25d41dbdacc341ee93c86f7a7f056123 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Advancing to term 2
W20240503 13:56:57.642882 15426 leader_election.cc:334] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:56:57.642993 15757 raft_consensus.cc:2454] T 25d41dbdacc341ee93c86f7a7f056123 P 76c386bef61a4bd5a907266a5118ffe2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 2.
I20240503 13:56:57.643349 15429 leader_election.cc:302] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61b887f0d40d497badeca5604c1f0546, 76c386bef61a4bd5a907266a5118ffe2; no voters: e47105a0f46e4511b7112e571688a4c7
I20240503 13:56:57.643595 25096 raft_consensus.cc:2781] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:56:57.643682 25096 raft_consensus.cc:686] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 [term 2 LEADER]: Becoming Leader. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Running, Role: LEADER
I20240503 13:56:57.643822 25096 consensus_queue.cc:237] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:57.646764 12907 catalog_manager.cc:5484] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 reported cstate change: term changed from 1 to 2, leader changed from e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) to 61b887f0d40d497badeca5604c1f0546 (127.6.98.69). New cstate: current_term: 2 leader_uuid: "61b887f0d40d497badeca5604c1f0546" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
W20240503 13:56:57.649327 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:56:57.663785 25016 raft_consensus.cc:482] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e47105a0f46e4511b7112e571688a4c7)
I20240503 13:56:57.663909 25016 raft_consensus.cc:504] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:57.664300 25016 leader_election.cc:288] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153)
I20240503 13:56:57.665069 23473 consensus_queue.cc:1035] T 6483eb56a18c41979d1faac1ef0b7816 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:57.665341 15492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a94e83461643d4adfc4abc7dfb74e7" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546" is_pre_election: true
I20240503 13:56:57.665468 15492 raft_consensus.cc:2454] T a6a94e83461643d4adfc4abc7dfb74e7 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 1.
W20240503 13:56:57.665692 13118 leader_election.cc:334] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:56:57.665781 13121 leader_election.cc:302] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61b887f0d40d497badeca5604c1f0546, 8ccf5a4ac450414c9d3579bd5800b770; no voters: e47105a0f46e4511b7112e571688a4c7
I20240503 13:56:57.665966 24831 raft_consensus.cc:2781] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 13:56:57.666014 24831 raft_consensus.cc:482] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e47105a0f46e4511b7112e571688a4c7)
I20240503 13:56:57.666049 24831 raft_consensus.cc:3037] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:56:57.667163 24831 raft_consensus.cc:504] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:57.667508 24831 leader_election.cc:288] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 2 election: Requested vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153)
I20240503 13:56:57.669255 24869 consensus_queue.cc:1035] T 6483eb56a18c41979d1faac1ef0b7816 P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 13:56:57.669723 13118 leader_election.cc:334] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:56:57.672276 15492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a94e83461643d4adfc4abc7dfb74e7" candidate_uuid: "8ccf5a4ac450414c9d3579bd5800b770" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546"
I20240503 13:56:57.672407 15492 raft_consensus.cc:3037] T a6a94e83461643d4adfc4abc7dfb74e7 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:56:57.673342 15492 raft_consensus.cc:2454] T a6a94e83461643d4adfc4abc7dfb74e7 P 61b887f0d40d497badeca5604c1f0546 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ccf5a4ac450414c9d3579bd5800b770 in term 2.
I20240503 13:56:57.673846 13121 leader_election.cc:302] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61b887f0d40d497badeca5604c1f0546, 8ccf5a4ac450414c9d3579bd5800b770; no voters: e47105a0f46e4511b7112e571688a4c7
I20240503 13:56:57.674048 25016 raft_consensus.cc:2781] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:56:57.674125 25016 raft_consensus.cc:686] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 [term 2 LEADER]: Becoming Leader. State: Replica: 8ccf5a4ac450414c9d3579bd5800b770, State: Running, Role: LEADER
I20240503 13:56:57.674240 25016 consensus_queue.cc:237] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:57.675875 25002 raft_consensus.cc:482] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e47105a0f46e4511b7112e571688a4c7)
I20240503 13:56:57.675980 25002 raft_consensus.cc:504] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:57.676340 25002 leader_election.cc:288] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:57.677368 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dca80c8a74a42c4b459315878ef5d14" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" is_pre_election: true
I20240503 13:56:57.677525 13445 raft_consensus.cc:2454] T 7dca80c8a74a42c4b459315878ef5d14 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 1.
I20240503 13:56:57.678180 12987 leader_election.cc:302] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 7690ee79577c45e3a2c8f8b552c0bff8; no voters:
I20240503 13:56:57.678349 24959 raft_consensus.cc:482] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e47105a0f46e4511b7112e571688a4c7)
I20240503 13:56:57.678442 24959 raft_consensus.cc:504] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:57.678649 12904 catalog_manager.cc:5484] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 reported cstate change: term changed from 1 to 2, leader changed from e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) to 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66). New cstate: current_term: 2 leader_uuid: "8ccf5a4ac450414c9d3579bd5800b770" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
I20240503 13:56:57.678807 24959 leader_election.cc:288] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
W20240503 13:56:57.679338 12987 leader_election.cc:334] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:56:57.679601 25002 raft_consensus.cc:2781] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 13:56:57.679653 25002 raft_consensus.cc:482] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Starting leader election (detected failure of leader e47105a0f46e4511b7112e571688a4c7)
I20240503 13:56:57.679690 25002 raft_consensus.cc:3037] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Advancing to term 2
W20240503 13:56:57.680130 15691 leader_election.cc:334] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:56:57.680634 13183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1fed738e1c46a6a0a0f9e2b9cfd9b0" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770" is_pre_election: true
I20240503 13:56:57.680752 13183 raft_consensus.cc:2454] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 1.
I20240503 13:56:57.680724 25002 raft_consensus.cc:504] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:57.681044 25002 leader_election.cc:288] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 2 election: Requested vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109)
I20240503 13:56:57.681092 15694 leader_election.cc:302] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 76c386bef61a4bd5a907266a5118ffe2, 8ccf5a4ac450414c9d3579bd5800b770; no voters: e47105a0f46e4511b7112e571688a4c7
I20240503 13:56:57.681651 24959 raft_consensus.cc:2781] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 13:56:57.681710 24959 raft_consensus.cc:482] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e47105a0f46e4511b7112e571688a4c7)
I20240503 13:56:57.681751 24959 raft_consensus.cc:3037] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:56:57.682804 24959 raft_consensus.cc:504] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:57.683092 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dca80c8a74a42c4b459315878ef5d14" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7690ee79577c45e3a2c8f8b552c0bff8"
I20240503 13:56:57.683153 24959 leader_election.cc:288] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 2 election: Requested vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), 8ccf5a4ac450414c9d3579bd5800b770 (127.6.98.66:36387)
I20240503 13:56:57.683228 13445 raft_consensus.cc:3037] T 7dca80c8a74a42c4b459315878ef5d14 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Advancing to term 2
W20240503 13:56:57.683844 12987 leader_election.cc:334] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:56:57.684293 13445 raft_consensus.cc:2454] T 7dca80c8a74a42c4b459315878ef5d14 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 2.
I20240503 13:56:57.684602 12987 leader_election.cc:302] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60ba3ffe3b7746469834cd91c84be14e, 7690ee79577c45e3a2c8f8b552c0bff8; no voters: e47105a0f46e4511b7112e571688a4c7
W20240503 13:56:57.684775 15691 leader_election.cc:334] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:56:57.684810 25002 raft_consensus.cc:2781] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:56:57.684793 13182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1fed738e1c46a6a0a0f9e2b9cfd9b0" candidate_uuid: "76c386bef61a4bd5a907266a5118ffe2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8ccf5a4ac450414c9d3579bd5800b770"
I20240503 13:56:57.684888 25002 raft_consensus.cc:686] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e [term 2 LEADER]: Becoming Leader. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Running, Role: LEADER
I20240503 13:56:57.684895 13182 raft_consensus.cc:3037] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 8ccf5a4ac450414c9d3579bd5800b770 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:56:57.685038 25002 consensus_queue.cc:237] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:57.685838 13182 raft_consensus.cc:2454] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 8ccf5a4ac450414c9d3579bd5800b770 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c386bef61a4bd5a907266a5118ffe2 in term 2.
I20240503 13:56:57.686111 15694 leader_election.cc:302] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 76c386bef61a4bd5a907266a5118ffe2, 8ccf5a4ac450414c9d3579bd5800b770; no voters: e47105a0f46e4511b7112e571688a4c7
I20240503 13:56:57.686371 24959 raft_consensus.cc:2781] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:56:57.686455 24959 raft_consensus.cc:686] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 [term 2 LEADER]: Becoming Leader. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Running, Role: LEADER
I20240503 13:56:57.686590 24959 consensus_queue.cc:237] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } }
I20240503 13:56:57.688406 12904 catalog_manager.cc:5484] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e reported cstate change: term changed from 1 to 2, leader changed from e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) to 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65). New cstate: current_term: 2 leader_uuid: "60ba3ffe3b7746469834cd91c84be14e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } }
I20240503 13:56:57.689060 23621 raft_consensus.cc:482] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e47105a0f46e4511b7112e571688a4c7)
I20240503 13:56:57.689162 23621 raft_consensus.cc:504] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:57.689538 23621 leader_election.cc:288] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:57.689875 12907 catalog_manager.cc:5484] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 reported cstate change: term changed from 1 to 2, leader changed from e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) to 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71). New cstate: current_term: 2 leader_uuid: "76c386bef61a4bd5a907266a5118ffe2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 } } }
I20240503 13:56:57.690405 13316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bdb9c14caf04abca39e56f8bcf1d1d4" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703" is_pre_election: true
I20240503 13:56:57.690531 13316 raft_consensus.cc:2454] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 1.
I20240503 13:56:57.690871 15561 leader_election.cc:302] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, ba0194098aa84449ae17e8a0d4c43703; no voters:
W20240503 13:56:57.692240 15558 leader_election.cc:334] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:56:57.692756 23621 raft_consensus.cc:2781] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 13:56:57.692814 23621 raft_consensus.cc:482] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Starting leader election (detected failure of leader e47105a0f46e4511b7112e571688a4c7)
I20240503 13:56:57.692852 23621 raft_consensus.cc:3037] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:56:57.693919 23621 raft_consensus.cc:504] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
I20240503 13:56:57.694311 23621 leader_election.cc:288] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 2 election: Requested vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67:42279)
I20240503 13:56:57.695472 13316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bdb9c14caf04abca39e56f8bcf1d1d4" candidate_uuid: "009f849ee7b54c80a3f5264695047e4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ba0194098aa84449ae17e8a0d4c43703"
I20240503 13:56:57.695595 13316 raft_consensus.cc:3037] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:56:57.696651 13316 raft_consensus.cc:2454] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P ba0194098aa84449ae17e8a0d4c43703 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009f849ee7b54c80a3f5264695047e4d in term 2.
I20240503 13:56:57.696954 15561 leader_election.cc:302] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:57.697695 23621 raft_consensus.cc:2781] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:56:57.697777 23621 raft_consensus.cc:686] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d [term 2 LEADER]: Becoming Leader. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Running, Role: LEADER
I20240503 13:56:57.697923 23621 consensus_queue.cc:237] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } }
W20240503 13:56:57.698144 15558 leader_election.cc:334] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:56:57.700137 12904 catalog_manager.cc:5484] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d reported cstate change: term changed from 1 to 2, leader changed from e47105a0f46e4511b7112e571688a4c7 (127.6.98.72) to 009f849ee7b54c80a3f5264695047e4d (127.6.98.70). New cstate: current_term: 2 leader_uuid: "009f849ee7b54c80a3f5264695047e4d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } }
I20240503 13:56:57.713279 12904 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 13:56:57.719094 15492 raft_consensus.cc:1261] T a21b33de70e94930bea00c0e16e713fc P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Refusing update from remote peer 7690ee79577c45e3a2c8f8b552c0bff8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:56:57.720005 24637 consensus_queue.cc:1035] T a21b33de70e94930bea00c0e16e713fc P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 13:56:57.720400 13380 consensus_peers.cc:485] T a21b33de70e94930bea00c0e16e713fc P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:57.723943 25344 tablet.cc:1722] T a21b33de70e94930bea00c0e16e713fc P 7690ee79577c45e3a2c8f8b552c0bff8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:57.724318 25344 tablet.cc:1620] T a21b33de70e94930bea00c0e16e713fc P 7690ee79577c45e3a2c8f8b552c0bff8: MemRowSet was empty: no flush needed.
I20240503 13:56:57.725589 12904 catalog_manager.cc:5576] TS 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153) does not have the latest schema for tablet a21b33de70e94930bea00c0e16e713fc (table rebalancer_dynamic_table [id=41dc105ceda04fabbfa4246ecf873aed]). Expected version 1 got 0
I20240503 13:56:57.727401 25210 tablet.cc:1722] T a21b33de70e94930bea00c0e16e713fc P 61b887f0d40d497badeca5604c1f0546: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 13:56:57.727722 25210 tablet.cc:1620] T a21b33de70e94930bea00c0e16e713fc P 61b887f0d40d497badeca5604c1f0546: MemRowSet was empty: no flush needed.
I20240503 13:56:57.736266 12904 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=41dc105ceda04fabbfa4246ecf873aed] alter complete (version 1)
I20240503 13:56:57.747284 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:56:57.751783 15473 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240503 13:56:57.756975 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:56:57.766906 13034 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 13:56:57.769760 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:56:57.770309 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:56:57.779172 12910 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_extra_table_74"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:56:57.800063 15473 tablet_service.cc:1467] Processing CreateTablet for tablet 9c207d545f984356bbf63ce2819d412c (DEFAULT_TABLE table=rebalancer_extra_table_74 [id=5121000a71434d439f7b22d37a9234f8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:57.800405 15473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c207d545f984356bbf63ce2819d412c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:57.800330 13034 tablet_service.cc:1467] Processing CreateTablet for tablet 9bef98a0c5e0411cb55141a93c62f5c0 (DEFAULT_TABLE table=rebalancer_extra_table_74 [id=5121000a71434d439f7b22d37a9234f8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:57.800660 13034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bef98a0c5e0411cb55141a93c62f5c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:57.802909 25280 tablet_bootstrap.cc:492] T 9c207d545f984356bbf63ce2819d412c P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:57.803755 13295 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240503 13:56:57.803912 25280 tablet_bootstrap.cc:654] T 9c207d545f984356bbf63ce2819d412c P 61b887f0d40d497badeca5604c1f0546: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:57.805658 15738 tablet_service.cc:1467] Processing CreateTablet for tablet 05ad17680502464ba4b195fa8b0c2dfc (DEFAULT_TABLE table=rebalancer_extra_table_74 [id=5121000a71434d439f7b22d37a9234f8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:57.805927 15738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05ad17680502464ba4b195fa8b0c2dfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:57.808537 15604 tablet_service.cc:1467] Processing CreateTablet for tablet 9c207d545f984356bbf63ce2819d412c (DEFAULT_TABLE table=rebalancer_extra_table_74 [id=5121000a71434d439f7b22d37a9234f8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:57.808808 15604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c207d545f984356bbf63ce2819d412c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:57.809109 13295 tablet_service.cc:1467] Processing CreateTablet for tablet 05ad17680502464ba4b195fa8b0c2dfc (DEFAULT_TABLE table=rebalancer_extra_table_74 [id=5121000a71434d439f7b22d37a9234f8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:57.809377 13295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05ad17680502464ba4b195fa8b0c2dfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:57.811774 15604 tablet_service.cc:1467] Processing CreateTablet for tablet 9bef98a0c5e0411cb55141a93c62f5c0 (DEFAULT_TABLE table=rebalancer_extra_table_74 [id=5121000a71434d439f7b22d37a9234f8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:57.812004 15604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bef98a0c5e0411cb55141a93c62f5c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:57.812896 13427 tablet_service.cc:1467] Processing CreateTablet for tablet 9c207d545f984356bbf63ce2819d412c (DEFAULT_TABLE table=rebalancer_extra_table_74 [id=5121000a71434d439f7b22d37a9234f8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:56:57.813155 13427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c207d545f984356bbf63ce2819d412c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:57.814872 25281 tablet_bootstrap.cc:492] T 9c207d545f984356bbf63ce2819d412c P 7690ee79577c45e3a2c8f8b552c0bff8: Bootstrap starting.
I20240503 13:56:57.815773 25281 tablet_bootstrap.cc:654] T 9c207d545f984356bbf63ce2819d412c P 7690ee79577c45e3a2c8f8b552c0bff8: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:57.817495 25281 tablet_bootstrap.cc:492] T 9c207d545f984356bbf63ce2819d412c P 7690ee79577c45e3a2c8f8b552c0bff8: No bootstrap required, opened a new log
I20240503 13:56:57.817577 25281 ts_tablet_manager.cc:1367] T 9c207d545f984356bbf63ce2819d412c P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:56:57.818058 25281 raft_consensus.cc:348] T 9c207d545f984356bbf63ce2819d412c P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:57.819787 25281 raft_consensus.cc:374] T 9c207d545f984356bbf63ce2819d412c P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:57.819830 25281 raft_consensus.cc:729] T 9c207d545f984356bbf63ce2819d412c P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Initialized, Role: FOLLOWER
I20240503 13:56:57.819947 25281 consensus_queue.cc:260] T 9c207d545f984356bbf63ce2819d412c P 7690ee79577c45e3a2c8f8b552c0bff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:57.820262 25281 ts_tablet_manager.cc:1398] T 9c207d545f984356bbf63ce2819d412c P 7690ee79577c45e3a2c8f8b552c0bff8: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:56:57.835876 25360 tablet_bootstrap.cc:492] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703: Bootstrap starting.
I20240503 13:56:57.836968 25280 tablet_bootstrap.cc:492] T 9c207d545f984356bbf63ce2819d412c P 61b887f0d40d497badeca5604c1f0546: No bootstrap required, opened a new log
I20240503 13:56:57.837080 25280 ts_tablet_manager.cc:1367] T 9c207d545f984356bbf63ce2819d412c P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.034s user 0.002s sys 0.000s
I20240503 13:56:57.837496 25280 raft_consensus.cc:348] T 9c207d545f984356bbf63ce2819d412c P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:57.837621 25280 raft_consensus.cc:374] T 9c207d545f984356bbf63ce2819d412c P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:57.837651 25280 raft_consensus.cc:729] T 9c207d545f984356bbf63ce2819d412c P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:57.837762 25280 consensus_queue.cc:260] T 9c207d545f984356bbf63ce2819d412c P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:57.838047 25280 ts_tablet_manager.cc:1398] T 9c207d545f984356bbf63ce2819d412c P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:56:57.839536 25358 tablet_bootstrap.cc:492] T 05ad17680502464ba4b195fa8b0c2dfc P 76c386bef61a4bd5a907266a5118ffe2: Bootstrap starting.
I20240503 13:56:57.839991 25359 tablet_bootstrap.cc:492] T 9c207d545f984356bbf63ce2819d412c P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:57.840845 25356 tablet_bootstrap.cc:492] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e: Bootstrap starting.
W20240503 13:56:57.842733 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:57.842782 25359 tablet_bootstrap.cc:654] T 9c207d545f984356bbf63ce2819d412c P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:57.844671 25358 tablet_bootstrap.cc:654] T 05ad17680502464ba4b195fa8b0c2dfc P 76c386bef61a4bd5a907266a5118ffe2: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:57.847179 25360 tablet_bootstrap.cc:654] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:57.848726 12904 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 13:56:57.849819 25356 tablet_bootstrap.cc:654] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:57.853960 25356 tablet_bootstrap.cc:492] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e: No bootstrap required, opened a new log
I20240503 13:56:57.854283 25356 ts_tablet_manager.cc:1367] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240503 13:56:57.854544 25358 tablet_bootstrap.cc:492] T 05ad17680502464ba4b195fa8b0c2dfc P 76c386bef61a4bd5a907266a5118ffe2: No bootstrap required, opened a new log
I20240503 13:56:57.855038 25358 ts_tablet_manager.cc:1367] T 05ad17680502464ba4b195fa8b0c2dfc P 76c386bef61a4bd5a907266a5118ffe2: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.002s
I20240503 13:56:57.855437 13316 raft_consensus.cc:1261] T db5568722b6344d18d6d6d271f05fc7a P ba0194098aa84449ae17e8a0d4c43703 [term 2 FOLLOWER]: Refusing update from remote peer 61b887f0d40d497badeca5604c1f0546: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 13:56:57.855453 25356 raft_consensus.cc:348] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:57.855562 25356 raft_consensus.cc:374] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:57.855605 25356 raft_consensus.cc:729] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Initialized, Role: FOLLOWER
I20240503 13:56:57.855711 25358 raft_consensus.cc:348] T 05ad17680502464ba4b195fa8b0c2dfc P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:57.855758 25356 consensus_queue.cc:260] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:57.856141 25356 ts_tablet_manager.cc:1398] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:57.857292 25358 raft_consensus.cc:374] T 05ad17680502464ba4b195fa8b0c2dfc P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:57.857362 25358 raft_consensus.cc:729] T 05ad17680502464ba4b195fa8b0c2dfc P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c386bef61a4bd5a907266a5118ffe2, State: Initialized, Role: FOLLOWER
I20240503 13:56:57.857509 25358 consensus_queue.cc:260] T 05ad17680502464ba4b195fa8b0c2dfc P 76c386bef61a4bd5a907266a5118ffe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:57.857827 25358 ts_tablet_manager.cc:1398] T 05ad17680502464ba4b195fa8b0c2dfc P 76c386bef61a4bd5a907266a5118ffe2: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 13:56:57.862038 25096 consensus_queue.cc:1035] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:56:57.864248 25359 tablet_bootstrap.cc:492] T 9c207d545f984356bbf63ce2819d412c P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:57.864455 25359 ts_tablet_manager.cc:1367] T 9c207d545f984356bbf63ce2819d412c P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.000s
I20240503 13:56:57.865097 25359 raft_consensus.cc:348] T 9c207d545f984356bbf63ce2819d412c P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:57.865407 25359 raft_consensus.cc:374] T 9c207d545f984356bbf63ce2819d412c P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:57.865762 25359 raft_consensus.cc:729] T 9c207d545f984356bbf63ce2819d412c P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:57.865967 25359 consensus_queue.cc:260] T 9c207d545f984356bbf63ce2819d412c P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:57.866362 25359 ts_tablet_manager.cc:1398] T 9c207d545f984356bbf63ce2819d412c P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:56:57.866638 25359 tablet_bootstrap.cc:492] T 9bef98a0c5e0411cb55141a93c62f5c0 P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:57.867736 25359 tablet_bootstrap.cc:654] T 9bef98a0c5e0411cb55141a93c62f5c0 P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:57.869040 25360 tablet_bootstrap.cc:492] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703: No bootstrap required, opened a new log
I20240503 13:56:57.869212 25360 ts_tablet_manager.cc:1367] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.000s
I20240503 13:56:57.869840 25360 raft_consensus.cc:348] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:57.870095 25360 raft_consensus.cc:374] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:57.870170 25360 raft_consensus.cc:729] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Initialized, Role: FOLLOWER
I20240503 13:56:57.870380 25360 consensus_queue.cc:260] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:57.870770 25360 ts_tablet_manager.cc:1398] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:57.871093 25344 tablet.cc:1722] T a21b33de70e94930bea00c0e16e713fc P 7690ee79577c45e3a2c8f8b552c0bff8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:57.871423 25344 tablet.cc:1620] T a21b33de70e94930bea00c0e16e713fc P 7690ee79577c45e3a2c8f8b552c0bff8: MemRowSet was empty: no flush needed.
I20240503 13:56:57.872339 25210 tablet.cc:1722] T a21b33de70e94930bea00c0e16e713fc P 61b887f0d40d497badeca5604c1f0546: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 13:56:57.872682 25210 tablet.cc:1620] T a21b33de70e94930bea00c0e16e713fc P 61b887f0d40d497badeca5604c1f0546: MemRowSet was empty: no flush needed.
I20240503 13:56:57.879318 12905 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table [id=41dc105ceda04fabbfa4246ecf873aed] alter complete (version 2)
I20240503 13:56:57.881320 25359 tablet_bootstrap.cc:492] T 9bef98a0c5e0411cb55141a93c62f5c0 P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:57.881436 25359 ts_tablet_manager.cc:1367] T 9bef98a0c5e0411cb55141a93c62f5c0 P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
W20240503 13:56:57.881610 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:57.881940 25359 raft_consensus.cc:348] T 9bef98a0c5e0411cb55141a93c62f5c0 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:57.882058 25359 raft_consensus.cc:374] T 9bef98a0c5e0411cb55141a93c62f5c0 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:57.882093 25359 raft_consensus.cc:729] T 9bef98a0c5e0411cb55141a93c62f5c0 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:57.882213 25359 consensus_queue.cc:260] T 9bef98a0c5e0411cb55141a93c62f5c0 P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:57.882493 25359 ts_tablet_manager.cc:1398] T 9bef98a0c5e0411cb55141a93c62f5c0 P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240503 13:56:57.883347 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:57.898463 24637 raft_consensus.cc:482] T 9c207d545f984356bbf63ce2819d412c P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:57.898594 24637 raft_consensus.cc:504] T 9c207d545f984356bbf63ce2819d412c P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:57.899055 24637 leader_election.cc:288] T 9c207d545f984356bbf63ce2819d412c P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951), 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153)
I20240503 13:56:57.899384 15624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c207d545f984356bbf63ce2819d412c" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d" is_pre_election: true
I20240503 13:56:57.899427 15492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c207d545f984356bbf63ce2819d412c" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546" is_pre_election: true
I20240503 13:56:57.899514 15624 raft_consensus.cc:2454] T 9c207d545f984356bbf63ce2819d412c P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 0.
I20240503 13:56:57.899559 15492 raft_consensus.cc:2454] T 9c207d545f984356bbf63ce2819d412c P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 0.
I20240503 13:56:57.899803 13380 leader_election.cc:302] T 9c207d545f984356bbf63ce2819d412c P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 7690ee79577c45e3a2c8f8b552c0bff8; no voters:
I20240503 13:56:57.900033 24637 raft_consensus.cc:2781] T 9c207d545f984356bbf63ce2819d412c P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:57.900080 24637 raft_consensus.cc:482] T 9c207d545f984356bbf63ce2819d412c P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:57.900112 24637 raft_consensus.cc:3037] T 9c207d545f984356bbf63ce2819d412c P 7690ee79577c45e3a2c8f8b552c0bff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:57.901047 24637 raft_consensus.cc:504] T 9c207d545f984356bbf63ce2819d412c P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:57.901422 24637 leader_election.cc:288] T 9c207d545f984356bbf63ce2819d412c P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 election: Requested vote from peers 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951), 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153)
I20240503 13:56:57.901708 15624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c207d545f984356bbf63ce2819d412c" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d"
I20240503 13:56:57.901824 15624 raft_consensus.cc:3037] T 9c207d545f984356bbf63ce2819d412c P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:57.901846 15492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c207d545f984356bbf63ce2819d412c" candidate_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b887f0d40d497badeca5604c1f0546"
I20240503 13:56:57.901993 15492 raft_consensus.cc:3037] T 9c207d545f984356bbf63ce2819d412c P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:57.902789 15624 raft_consensus.cc:2454] T 9c207d545f984356bbf63ce2819d412c P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 1.
I20240503 13:56:57.902804 15492 raft_consensus.cc:2454] T 9c207d545f984356bbf63ce2819d412c P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7690ee79577c45e3a2c8f8b552c0bff8 in term 1.
I20240503 13:56:57.903067 13383 leader_election.cc:302] T 9c207d545f984356bbf63ce2819d412c P 7690ee79577c45e3a2c8f8b552c0bff8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61b887f0d40d497badeca5604c1f0546, 7690ee79577c45e3a2c8f8b552c0bff8; no voters:
I20240503 13:56:57.903286 24637 raft_consensus.cc:2781] T 9c207d545f984356bbf63ce2819d412c P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:57.903371 24637 raft_consensus.cc:686] T 9c207d545f984356bbf63ce2819d412c P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 LEADER]: Becoming Leader. State: Replica: 7690ee79577c45e3a2c8f8b552c0bff8, State: Running, Role: LEADER
I20240503 13:56:57.903506 24637 consensus_queue.cc:237] T 9c207d545f984356bbf63ce2819d412c P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } }
I20240503 13:56:57.906014 12904 catalog_manager.cc:5484] T 9c207d545f984356bbf63ce2819d412c P 7690ee79577c45e3a2c8f8b552c0bff8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68). New cstate: current_term: 1 leader_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } }
W20240503 13:56:57.946112 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:57.958580 13316 raft_consensus.cc:1261] T 3a140864f59b483bb5fea3de83d28ecf P ba0194098aa84449ae17e8a0d4c43703 [term 2 FOLLOWER]: Refusing update from remote peer 009f849ee7b54c80a3f5264695047e4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 13:56:57.959024 24901 consensus_queue.cc:1035] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 13:56:57.962404 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:57.963675 13445 raft_consensus.cc:1261] T 0f9224c2e3b946d993a5fa5c95a1d681 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 2 FOLLOWER]: Refusing update from remote peer 76c386bef61a4bd5a907266a5118ffe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 13:56:57.964131 24959 consensus_queue.cc:1035] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:57.964346 25002 raft_consensus.cc:482] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:57.964460 25002 raft_consensus.cc:504] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:57.964919 25002 leader_election.cc:288] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:57.965850 15624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bef98a0c5e0411cb55141a93c62f5c0" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d" is_pre_election: true
I20240503 13:56:57.965911 13446 raft_consensus.cc:1261] T f58fc3bc579748bb88de34612808ccda P 7690ee79577c45e3a2c8f8b552c0bff8 [term 2 FOLLOWER]: Refusing update from remote peer 60ba3ffe3b7746469834cd91c84be14e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 13:56:57.965988 15624 raft_consensus.cc:2454] T 9bef98a0c5e0411cb55141a93c62f5c0 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 0.
I20240503 13:56:57.966801 12987 leader_election.cc:302] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 60ba3ffe3b7746469834cd91c84be14e; no voters:
I20240503 13:56:57.966796 25002 consensus_queue.cc:1035] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 13:56:57.966979 12987 leader_election.cc:334] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:56:57.967063 25002 raft_consensus.cc:2781] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:57.967134 25002 raft_consensus.cc:482] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:57.967175 25002 raft_consensus.cc:3037] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:57.968420 25002 raft_consensus.cc:504] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:57.968796 25002 leader_election.cc:288] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 election: Requested vote from peers 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:57.970196 15624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bef98a0c5e0411cb55141a93c62f5c0" candidate_uuid: "60ba3ffe3b7746469834cd91c84be14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d"
I20240503 13:56:57.970403 15624 raft_consensus.cc:3037] T 9bef98a0c5e0411cb55141a93c62f5c0 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
W20240503 13:56:57.971748 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 13:56:57.971832 12987 leader_election.cc:334] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:56:57.972023 15624 raft_consensus.cc:2454] T 9bef98a0c5e0411cb55141a93c62f5c0 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ba3ffe3b7746469834cd91c84be14e in term 1.
I20240503 13:56:57.972568 12987 leader_election.cc:302] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 60ba3ffe3b7746469834cd91c84be14e; no voters: e47105a0f46e4511b7112e571688a4c7
I20240503 13:56:57.972843 25373 raft_consensus.cc:2781] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:57.972954 25373 raft_consensus.cc:686] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e [term 1 LEADER]: Becoming Leader. State: Replica: 60ba3ffe3b7746469834cd91c84be14e, State: Running, Role: LEADER
I20240503 13:56:57.973232 25373 consensus_queue.cc:237] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } }
I20240503 13:56:57.974778 15757 raft_consensus.cc:1261] T 10ee7dee14364389871900670a02906a P 76c386bef61a4bd5a907266a5118ffe2 [term 2 FOLLOWER]: Refusing update from remote peer 009f849ee7b54c80a3f5264695047e4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 13:56:57.976457 24901 consensus_queue.cc:1035] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:56:57.977865 12905 catalog_manager.cc:5484] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e reported cstate change: term changed from 0 to 1, leader changed from <none> to 60ba3ffe3b7746469834cd91c84be14e (127.6.98.65). New cstate: current_term: 1 leader_uuid: "60ba3ffe3b7746469834cd91c84be14e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 } } }
W20240503 13:56:57.982553 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:57.986609 12904 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 13:56:57.992609 25344 tablet.cc:1722] T a21b33de70e94930bea00c0e16e713fc P 7690ee79577c45e3a2c8f8b552c0bff8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:57.992895 25210 tablet.cc:1722] T a21b33de70e94930bea00c0e16e713fc P 61b887f0d40d497badeca5604c1f0546: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 13:56:57.997805 12904 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: rebalancer_dynamic_table_new_name [id=41dc105ceda04fabbfa4246ecf873aed] alter complete (version 3)
I20240503 13:56:58.000054 15757 raft_consensus.cc:1261] T eb92a057165f4da7b96d910d847ff761 P 76c386bef61a4bd5a907266a5118ffe2 [term 2 FOLLOWER]: Refusing update from remote peer 61b887f0d40d497badeca5604c1f0546: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 13:56:58.000569 20722 consensus_queue.cc:1035] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:58.001149 15492 raft_consensus.cc:1261] T 5f34ac73ebe544a1b7a34e8c2d11568e P 61b887f0d40d497badeca5604c1f0546 [term 2 FOLLOWER]: Refusing update from remote peer 7690ee79577c45e3a2c8f8b552c0bff8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 13:56:58.001845 25376 consensus_queue.cc:1035] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 13:56:58.004122 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 13:56:58.006642 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 13:56:58.052563 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:58.053104 15757 raft_consensus.cc:1261] T 4b4b26adad184e1ea57c00a44635679c P 76c386bef61a4bd5a907266a5118ffe2 [term 2 FOLLOWER]: Refusing update from remote peer 7690ee79577c45e3a2c8f8b552c0bff8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 13:56:58.053498 25376 consensus_queue.cc:1035] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 13:56:58.063766 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:58.075449 13316 raft_consensus.cc:1261] T 557f14aaf5564ebaa461720a4d7997ff P ba0194098aa84449ae17e8a0d4c43703 [term 2 FOLLOWER]: Refusing update from remote peer 009f849ee7b54c80a3f5264695047e4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 13:56:58.075891 25269 consensus_queue.cc:1035] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:56:58.102759 12905 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:58.102981 12905 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46808:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 13:56:58.105773 12905 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f21c260fa03f4452870a21b3091e1130: Sending DeleteTablet for 3 replicas of tablet a21b33de70e94930bea00c0e16e713fc
I20240503 13:56:58.108661 13427 tablet_service.cc:1514] Processing DeleteTablet for tablet a21b33de70e94930bea00c0e16e713fc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:58 UTC) from {username='slave'} at 127.0.0.1:33678
I20240503 13:56:58.109037 25397 tablet_replica.cc:330] T a21b33de70e94930bea00c0e16e713fc P 7690ee79577c45e3a2c8f8b552c0bff8: stopping tablet replica
I20240503 13:56:58.109182 25397 raft_consensus.cc:2229] T a21b33de70e94930bea00c0e16e713fc P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:56:58.109426 25397 raft_consensus.cc:2258] T a21b33de70e94930bea00c0e16e713fc P 7690ee79577c45e3a2c8f8b552c0bff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:58.109488 13051 raft_consensus.cc:1261] T dd39f3ab7cdb42b38267214056605328 P 60ba3ffe3b7746469834cd91c84be14e [term 2 FOLLOWER]: Refusing update from remote peer 009f849ee7b54c80a3f5264695047e4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 13:56:58.109795 15473 tablet_service.cc:1514] Processing DeleteTablet for tablet a21b33de70e94930bea00c0e16e713fc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 13:56:58 UTC) from {username='slave'} at 127.0.0.1:58758
I20240503 13:56:58.110126 25269 consensus_queue.cc:1035] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba3ffe3b7746469834cd91c84be14e" member_type: VOTER last_known_addr { host: "127.6.98.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:56:58.110440 25397 ts_tablet_manager.cc:1875] T a21b33de70e94930bea00c0e16e713fc P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 13:56:58.115998 13445 raft_consensus.cc:1261] T 7dca80c8a74a42c4b459315878ef5d14 P 7690ee79577c45e3a2c8f8b552c0bff8 [term 2 FOLLOWER]: Refusing update from remote peer 60ba3ffe3b7746469834cd91c84be14e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 13:56:58.116601 25002 consensus_queue.cc:1035] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7690ee79577c45e3a2c8f8b552c0bff8" member_type: VOTER last_known_addr { host: "127.6.98.68" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:56:58.120606 25397 ts_tablet_manager.cc:1888] T a21b33de70e94930bea00c0e16e713fc P 7690ee79577c45e3a2c8f8b552c0bff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:58.120762 25397 log.cc:1196] T a21b33de70e94930bea00c0e16e713fc P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-3/wal/wals/a21b33de70e94930bea00c0e16e713fc
I20240503 13:56:58.121222 25397 ts_tablet_manager.cc:1909] T a21b33de70e94930bea00c0e16e713fc P 7690ee79577c45e3a2c8f8b552c0bff8: Deleting consensus metadata
W20240503 13:56:58.121958 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:58.125257 12896 catalog_manager.cc:4830] TS 7690ee79577c45e3a2c8f8b552c0bff8 (127.6.98.68:36109): tablet a21b33de70e94930bea00c0e16e713fc (table rebalancer_dynamic_table_new_name [id=41dc105ceda04fabbfa4246ecf873aed]) successfully deleted
I20240503 13:56:58.129027 25398 tablet_replica.cc:330] T a21b33de70e94930bea00c0e16e713fc P 61b887f0d40d497badeca5604c1f0546: stopping tablet replica
W20240503 13:56:58.132321 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:58.132659 25398 raft_consensus.cc:2229] T a21b33de70e94930bea00c0e16e713fc P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:56:58.132942 25398 raft_consensus.cc:2258] T a21b33de70e94930bea00c0e16e713fc P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:56:58.134232 15624 raft_consensus.cc:1261] T d3849e4effe64656bd8b0f504c9dc17a P 009f849ee7b54c80a3f5264695047e4d [term 2 FOLLOWER]: Refusing update from remote peer 8ccf5a4ac450414c9d3579bd5800b770: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 13:56:58.134379 25398 ts_tablet_manager.cc:1875] T a21b33de70e94930bea00c0e16e713fc P 61b887f0d40d497badeca5604c1f0546: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 13:56:58.134927 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:56:58.135326 25016 consensus_queue.cc:1035] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:56:58.136857 25398 ts_tablet_manager.cc:1888] T a21b33de70e94930bea00c0e16e713fc P 61b887f0d40d497badeca5604c1f0546: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 13:56:58.137006 25398 log.cc:1196] T a21b33de70e94930bea00c0e16e713fc P 61b887f0d40d497badeca5604c1f0546: Deleting WAL directory at /tmp/dist-test-taskFG4o4C/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714744538398999-6537-0/raft_consensus-itest-cluster/ts-4/wal/wals/a21b33de70e94930bea00c0e16e713fc
I20240503 13:56:58.137316 25398 ts_tablet_manager.cc:1909] T a21b33de70e94930bea00c0e16e713fc P 61b887f0d40d497badeca5604c1f0546: Deleting consensus metadata
I20240503 13:56:58.141044 12899 catalog_manager.cc:4830] TS 61b887f0d40d497badeca5604c1f0546 (127.6.98.69:39153): tablet a21b33de70e94930bea00c0e16e713fc (table rebalancer_dynamic_table_new_name [id=41dc105ceda04fabbfa4246ecf873aed]) successfully deleted
I20240503 13:56:58.144130 25254 raft_consensus.cc:482] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:58.144263 25254 raft_consensus.cc:504] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:58.144757 25254 leader_election.cc:288] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885)
I20240503 13:56:58.145547 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05ad17680502464ba4b195fa8b0c2dfc" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2" is_pre_election: true
I20240503 13:56:58.145727 15757 raft_consensus.cc:2454] T 05ad17680502464ba4b195fa8b0c2dfc P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 0.
I20240503 13:56:58.146178 13252 leader_election.cc:302] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76c386bef61a4bd5a907266a5118ffe2, ba0194098aa84449ae17e8a0d4c43703; no voters:
W20240503 13:56:58.146405 13249 leader_election.cc:334] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:56:58.146658 25254 raft_consensus.cc:2781] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:58.146710 25254 raft_consensus.cc:482] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:58.146747 25254 raft_consensus.cc:3037] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:58.148029 25254 raft_consensus.cc:504] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
I20240503 13:56:58.148398 25254 leader_election.cc:288] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: Requested vote from peers e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753), 76c386bef61a4bd5a907266a5118ffe2 (127.6.98.71:42885)
I20240503 13:56:58.149721 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05ad17680502464ba4b195fa8b0c2dfc" candidate_uuid: "ba0194098aa84449ae17e8a0d4c43703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c386bef61a4bd5a907266a5118ffe2"
I20240503 13:56:58.149861 15757 raft_consensus.cc:3037] T 05ad17680502464ba4b195fa8b0c2dfc P 76c386bef61a4bd5a907266a5118ffe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:58.151096 15757 raft_consensus.cc:2454] T 05ad17680502464ba4b195fa8b0c2dfc P 76c386bef61a4bd5a907266a5118ffe2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0194098aa84449ae17e8a0d4c43703 in term 1.
I20240503 13:56:58.151525 13252 leader_election.cc:302] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76c386bef61a4bd5a907266a5118ffe2, ba0194098aa84449ae17e8a0d4c43703; no voters:
I20240503 13:56:58.152307 24869 raft_consensus.cc:2781] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:58.152400 24869 raft_consensus.cc:686] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 [term 1 LEADER]: Becoming Leader. State: Replica: ba0194098aa84449ae17e8a0d4c43703, State: Running, Role: LEADER
I20240503 13:56:58.152566 24869 consensus_queue.cc:237] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } }
W20240503 13:56:58.154016 13249 leader_election.cc:334] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:56:58.155226 12912 catalog_manager.cc:5484] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 reported cstate change: term changed from 0 to 1, leader changed from <none> to ba0194098aa84449ae17e8a0d4c43703 (127.6.98.67). New cstate: current_term: 1 leader_uuid: "ba0194098aa84449ae17e8a0d4c43703" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } peers { permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 } } peers { permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 } } }
I20240503 13:56:58.167446 15492 raft_consensus.cc:1261] T a6a94e83461643d4adfc4abc7dfb74e7 P 61b887f0d40d497badeca5604c1f0546 [term 2 FOLLOWER]: Refusing update from remote peer 8ccf5a4ac450414c9d3579bd5800b770: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 13:56:58.168283 24831 consensus_queue.cc:1035] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:56:58.173337 15757 raft_consensus.cc:1261] T 25d41dbdacc341ee93c86f7a7f056123 P 76c386bef61a4bd5a907266a5118ffe2 [term 2 FOLLOWER]: Refusing update from remote peer 61b887f0d40d497badeca5604c1f0546: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 13:56:58.174273 25096 consensus_queue.cc:1035] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 13:56:58.179075 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:58.211377 12911 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46808:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 13:56:58.217649 15473 tablet_service.cc:1467] Processing CreateTablet for tablet 7519f1091c6e4bdda68e8c7bafcff9f5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f385c193cca7407884caa77a353b3420]), partition=
I20240503 13:56:58.217914 15473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7519f1091c6e4bdda68e8c7bafcff9f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:58.218822 15604 tablet_service.cc:1467] Processing CreateTablet for tablet 7519f1091c6e4bdda68e8c7bafcff9f5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f385c193cca7407884caa77a353b3420]), partition=
I20240503 13:56:58.219074 15604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7519f1091c6e4bdda68e8c7bafcff9f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:56:58.221726 25359 tablet_bootstrap.cc:492] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 009f849ee7b54c80a3f5264695047e4d: Bootstrap starting.
I20240503 13:56:58.222705 25359 tablet_bootstrap.cc:654] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 009f849ee7b54c80a3f5264695047e4d: Neither blocks nor log segments found. Creating new log.
W20240503 13:56:58.225322 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:58.231077 25280 tablet_bootstrap.cc:492] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546: Bootstrap starting.
I20240503 13:56:58.232059 25280 tablet_bootstrap.cc:654] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546: Neither blocks nor log segments found. Creating new log.
I20240503 13:56:58.232585 25359 tablet_bootstrap.cc:492] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 009f849ee7b54c80a3f5264695047e4d: No bootstrap required, opened a new log
I20240503 13:56:58.232668 25359 ts_tablet_manager.cc:1367] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 009f849ee7b54c80a3f5264695047e4d: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 13:56:58.233147 25280 tablet_bootstrap.cc:492] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546: No bootstrap required, opened a new log
I20240503 13:56:58.233120 25359 raft_consensus.cc:348] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:58.233212 25359 raft_consensus.cc:374] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:58.233225 25280 ts_tablet_manager.cc:1367] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:56:58.233242 25359 raft_consensus.cc:729] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009f849ee7b54c80a3f5264695047e4d, State: Initialized, Role: FOLLOWER
I20240503 13:56:58.233363 25359 consensus_queue.cc:260] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 009f849ee7b54c80a3f5264695047e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:58.233629 25359 ts_tablet_manager.cc:1398] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 009f849ee7b54c80a3f5264695047e4d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:56:58.233640 25280 raft_consensus.cc:348] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:58.233737 25280 raft_consensus.cc:374] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:56:58.233768 25280 raft_consensus.cc:729] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Initialized, Role: FOLLOWER
I20240503 13:56:58.233877 25280 consensus_queue.cc:260] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:58.234171 25280 ts_tablet_manager.cc:1398] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240503 13:56:58.249507 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:58.256976 13183 raft_consensus.cc:1261] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 8ccf5a4ac450414c9d3579bd5800b770 [term 2 FOLLOWER]: Refusing update from remote peer 76c386bef61a4bd5a907266a5118ffe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 13:56:58.257510 24959 consensus_queue.cc:1035] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccf5a4ac450414c9d3579bd5800b770" member_type: VOTER last_known_addr { host: "127.6.98.66" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:56:58.266558 13316 raft_consensus.cc:1261] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P ba0194098aa84449ae17e8a0d4c43703 [term 2 FOLLOWER]: Refusing update from remote peer 009f849ee7b54c80a3f5264695047e4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 13:56:58.267045 23621 consensus_queue.cc:1035] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0194098aa84449ae17e8a0d4c43703" member_type: VOTER last_known_addr { host: "127.6.98.67" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:56:58.304813 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:56:58.306645 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:56:58.306661 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:56:58.309069 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:56:58.316236 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:56:58.332837 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 13:56:58.343672 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:56:58.345149 15473 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240503 13:56:58.365298 25096 raft_consensus.cc:482] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:56:58.365427 25096 raft_consensus.cc:504] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:58.365825 25096 leader_election.cc:288] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
I20240503 13:56:58.366801 15623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7519f1091c6e4bdda68e8c7bafcff9f5" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d" is_pre_election: true
I20240503 13:56:58.366971 15623 raft_consensus.cc:2454] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 0.
I20240503 13:56:58.367280 15426 leader_election.cc:302] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 61b887f0d40d497badeca5604c1f0546; no voters:
W20240503 13:56:58.367559 15426 leader_election.cc:334] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:56:58.368028 25096 raft_consensus.cc:2781] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:56:58.368088 25096 raft_consensus.cc:482] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:56:58.368121 25096 raft_consensus.cc:3037] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:58.374104 25096 raft_consensus.cc:504] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
I20240503 13:56:58.374503 25096 leader_election.cc:288] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 election: Requested vote from peers 009f849ee7b54c80a3f5264695047e4d (127.6.98.70:34951), e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753)
W20240503 13:56:58.375656 15426 leader_election.cc:334] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:56:58.376469 15623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7519f1091c6e4bdda68e8c7bafcff9f5" candidate_uuid: "61b887f0d40d497badeca5604c1f0546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009f849ee7b54c80a3f5264695047e4d"
I20240503 13:56:58.376622 15623 raft_consensus.cc:3037] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 009f849ee7b54c80a3f5264695047e4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:56:58.377660 15623 raft_consensus.cc:2454] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 009f849ee7b54c80a3f5264695047e4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61b887f0d40d497badeca5604c1f0546 in term 1.
I20240503 13:56:58.378082 25002 consensus_queue.cc:1035] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:58.378114 15426 leader_election.cc:302] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 009f849ee7b54c80a3f5264695047e4d, 61b887f0d40d497badeca5604c1f0546; no voters: e47105a0f46e4511b7112e571688a4c7
I20240503 13:56:58.379165 25096 raft_consensus.cc:2781] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:56:58.379243 25096 raft_consensus.cc:686] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 [term 1 LEADER]: Becoming Leader. State: Replica: 61b887f0d40d497badeca5604c1f0546, State: Running, Role: LEADER
I20240503 13:56:58.379359 25096 consensus_queue.cc:237] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } }
W20240503 13:56:58.382251 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 13:56:58.385653 12906 catalog_manager.cc:5484] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61b887f0d40d497badeca5604c1f0546 (127.6.98.69). New cstate: current_term: 1 leader_uuid: "61b887f0d40d497badeca5604c1f0546" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 } } peers { permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 } } peers { permanent_uuid: "e47105a0f46e4511b7112e571688a4c7" member_type: VOTER last_known_addr { host: "127.6.98.72" port: 44753 } } }
I20240503 13:56:58.420168 25376 consensus_queue.cc:1035] T 9c207d545f984356bbf63ce2819d412c P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:56:58.424024 25376 consensus_queue.cc:1035] T 9c207d545f984356bbf63ce2819d412c P 7690ee79577c45e3a2c8f8b552c0bff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b887f0d40d497badeca5604c1f0546" member_type: VOTER last_known_addr { host: "127.6.98.69" port: 39153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 13:56:58.453377 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 13:56:58.524278 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 13:56:58.553228 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 13:56:58.586186 12896 catalog_manager.cc:4573] TS e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): CreateTablet RPC failed for tablet 7519f1091c6e4bdda68e8c7bafcff9f5: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:56:58.601110 25254 consensus_queue.cc:1035] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c386bef61a4bd5a907266a5118ffe2" member_type: VOTER last_known_addr { host: "127.6.98.71" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 13:56:58.605747 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 13:56:58.614856 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 13:56:58.668335 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 13:56:58.715063 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 13:56:58.717273 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 13:56:58.738013 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:56:58.745729 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:56:58.746405 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:56:58.746825 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:56:58.752996 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:56:58.755437 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:56:58.768020 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 13:56:58.773012 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:56:58.779464 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 13:56:58.781347 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:56:58.832176 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 13:56:58.853549 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 13:56:58.853642 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 13:56:58.859709 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 13:56:58.870345 25368 consensus_queue.cc:1035] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009f849ee7b54c80a3f5264695047e4d" member_type: VOTER last_known_addr { host: "127.6.98.70" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 13:56:58.874145 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 13:56:58.876614 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 13:56:58.877293 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 13:56:58.877454 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 13:56:58.891894 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 13:56:58.892982 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 13:56:58.921986 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 13:56:58.922731 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 13:56:58.951056 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 13:56:58.973261 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 13:56:58.980899 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 13:56:58.987023 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 13:56:58.992786 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 13:56:59.000977 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 13:56:59.015573 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 13:56:59.021958 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 13:56:59.030483 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 13:56:59.055776 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 13:56:59.063980 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 13:56:59.083102 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 13:56:59.110745 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 13:56:59.124816 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 13:56:59.128250 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 13:56:59.129204 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 13:56:59.141969 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 13:56:59.192750 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 13:56:59.193420 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 13:56:59.197546 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:56:59.198376 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:56:59.201766 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:56:59.209102 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:56:59.213783 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:56:59.217531 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 13:56:59.226573 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 13:56:59.260301 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:56:59.329375 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:56:59.493512 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 13:56:59.631314 12896 catalog_manager.cc:4573] TS e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2427783ee0e348b98453818704ece5f8: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:56:59.654523 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:56:59.654645 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:56:59.656787 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:56:59.661475 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:56:59.663827 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:56:59.664047 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:56:59.671831 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:56:59.768497 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:00.027698 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 13:57:00.067474 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:00.073938 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:00.076277 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:00.083153 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:00.083483 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:00.085422 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:00.085489 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:57:00.140754 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 13:57:00.234058 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 13:57:00.241310 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:00.297686 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 13:57:00.304179 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 13:57:00.422926 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 13:57:00.444864 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 13:57:00.461927 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 13:57:00.466166 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 13:57:00.474494 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 13:57:00.476480 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 13:57:00.494216 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 13:57:00.503201 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:00.513263 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:00.517192 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:00.517412 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:57:00.529992 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 13:57:00.537382 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:00.539768 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:00.543718 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:57:00.633734 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:00.647779 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 13:57:00.684882 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 13:57:00.716120 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 13:57:00.717351 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 13:57:00.805754 12896 catalog_manager.cc:4573] TS e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): CreateTablet RPC failed for tablet 2b362fed86c042868fc5b2fbaf83b369: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:00.846035 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 13:57:00.921684 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 13:57:00.947559 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:00.952816 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:00.960898 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:00.961578 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:57:00.973140 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 13:57:00.977770 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:00.978394 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:00.985328 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:57:01.000591 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 13:57:01.046049 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:01.093165 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 13:57:01.126361 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 13:57:01.135630 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 13:57:01.139693 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 13:57:01.149600 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 13:57:01.157341 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 13:57:01.178267 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 13:57:01.207058 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 13:57:01.242990 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 13:57:01.272063 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 13:57:01.273804 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 13:57:01.302367 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 13:57:01.321192 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 13:57:01.349094 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 13:57:01.355600 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 13:57:01.369597 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:01.375787 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:57:01.376065 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 13:57:01.390290 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:01.391867 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:01.392861 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:01.393308 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:01.394191 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:57:01.396119 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 13:57:01.400665 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 13:57:01.412019 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 13:57:01.416536 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 13:57:01.444541 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 13:57:01.451835 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 13:57:01.467209 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 13:57:01.468324 13118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240503 13:57:01.469046 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 13:57:01.510560 15558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240503 13:57:01.511313 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 13:57:01.528321 15691 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 118 similar messages]
W20240503 13:57:01.529284 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 13:57:01.529284 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 13:57:01.530308 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 13:57:01.533077 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 77 similar messages]
W20240503 13:57:01.545315 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 13:57:01.547757 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:01.549806 12987 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240503 13:57:01.570323 15426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 124 similar messages]
W20240503 13:57:01.585158 13249 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 123 similar messages]
W20240503 13:57:01.585924 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 13:57:01.601649 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 13:57:01.623289 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 13:57:01.640084 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 13:57:01.647162 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 13:57:01.675537 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 13:57:01.701468 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 13:57:01.708037 12896 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240503 13:57:01.720712 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 13:57:01.762538 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 13:57:01.806962 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 13:57:01.807250 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 13:57:01.813069 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:01.818039 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:01.823814 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:01.826570 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:01.827749 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:57:01.828279 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 13:57:01.836239 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 13:57:01.853796 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:57:01.992949 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:02.099016 12896 catalog_manager.cc:4573] TS e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): CreateTablet RPC failed for tablet 9bef98a0c5e0411cb55141a93c62f5c0: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:57:02.253716 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:02.255112 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:02.258368 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:02.259322 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:02.264045 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:02.271546 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:02.302839 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:57:02.511593 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:02.532589 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 13:57:02.671916 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:02.674289 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:02.684334 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:02.687160 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:02.687464 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:02.689200 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:02.694394 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:57:02.761849 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 13:57:02.809139 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 13:57:02.814177 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 13:57:02.833905 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 13:57:02.834513 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 13:57:02.855974 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 13:57:02.941689 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 13:57:02.966571 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 13:57:02.983135 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 13:57:02.998976 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 13:57:03.015753 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:03.026444 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 13:57:03.065974 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 13:57:03.102841 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:03.106127 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:03.110698 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:03.111011 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:03.121088 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:03.121721 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:03.149468 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:57:03.161064 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 13:57:03.210726 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 13:57:03.250939 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 13:57:03.271610 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 13:57:03.387324 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 13:57:03.403841 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 13:57:03.413550 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 13:57:03.449373 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:03.466713 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 13:57:03.542527 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 13:57:03.546226 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:03.552188 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:03.555150 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:03.558405 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:03.560273 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:57:03.576654 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 13:57:03.583958 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:57:03.584702 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 13:57:03.601027 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:57:03.611255 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 13:57:03.611297 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 13:57:03.636065 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 13:57:03.649466 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 13:57:03.720417 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 13:57:03.731726 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 13:57:03.751852 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 13:57:03.781549 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 13:57:03.783339 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 13:57:03.785120 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 13:57:03.786121 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 13:57:03.817822 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 13:57:03.822969 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 13:57:03.846060 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 13:57:03.862982 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 13:57:03.866045 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 13:57:03.876945 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 13:57:03.884425 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 13:57:03.931413 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 13:57:03.980576 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 13:57:03.981863 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 13:57:03.987648 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:03.988004 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:03.991665 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:57:04.000741 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 13:57:04.003937 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:57:04.004367 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 13:57:04.010679 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:57:04.014210 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:04.022859 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:57:04.025431 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 13:57:04.030076 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:57:04.058805 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 13:57:04.087539 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 13:57:04.098412 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 13:57:04.126063 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 13:57:04.202831 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 13:57:04.209975 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 13:57:04.234534 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 13:57:04.234443 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 13:57:04.253895 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 13:57:04.270262 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 13:57:04.274132 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 13:57:04.290385 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 13:57:04.349462 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 13:57:04.401525 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 13:57:04.453354 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:04.454896 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:04.456405 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:04.459220 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:04.468637 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:57:04.472672 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:57:04.473618 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:04.478960 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:57:04.853972 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:57:04.869273 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:04.871039 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:04.872761 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:04.888890 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:04.894804 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:04.911450 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:04.911478 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:57:04.936077 12896 catalog_manager.cc:4573] TS e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): CreateTablet RPC failed for tablet 2b362fed86c042868fc5b2fbaf83b369: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:05.040684 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 13:57:05.192313 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 13:57:05.193058 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 13:57:05.215713 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 13:57:05.296854 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 13:57:05.303934 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:05.311316 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:05.315894 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:57:05.336977 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 13:57:05.339057 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:05.341394 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:05.341826 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:05.348142 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:57:05.359720 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:05.384600 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 13:57:05.402220 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 13:57:05.455706 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 13:57:05.479574 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 13:57:05.491029 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 13:57:05.565086 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 13:57:05.617921 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 13:57:05.735630 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 13:57:05.740164 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 13:57:05.740960 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:05.741827 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:05.749416 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:05.753549 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:05.757114 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:05.767129 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:05.775893 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:57:05.785519 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:05.810745 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 13:57:05.824050 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 13:57:05.856542 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 13:57:05.885721 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 13:57:05.930964 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 13:57:05.963377 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 13:57:05.970180 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 13:57:06.012727 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 13:57:06.019414 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 13:57:06.049243 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 13:57:06.054075 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 13:57:06.162636 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 13:57:06.165992 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:57:06.175951 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:06.176620 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:06.176913 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:06.196930 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:06.197599 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:57:06.214716 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 13:57:06.218214 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:57:06.224725 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 13:57:06.241799 12896 catalog_manager.cc:4573] TS e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): CreateTablet RPC failed for tablet 9bef98a0c5e0411cb55141a93c62f5c0: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:57:06.245061 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:57:06.256636 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 13:57:06.326609 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 13:57:06.355581 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 13:57:06.355746 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 13:57:06.383745 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 13:57:06.387545 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 13:57:06.413345 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 13:57:06.430958 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 13:57:06.436368 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 13:57:06.441587 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 13:57:06.441707 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 13:57:06.453297 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 13:57:06.456012 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 13:57:06.471673 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 13:57:06.482673 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 13:57:06.487020 13118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240503 13:57:06.498577 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 13:57:06.526168 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 13:57:06.535770 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240503 13:57:06.552968 15691 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 109 similar messages]
W20240503 13:57:06.580425 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 13:57:06.584136 15426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 121 similar messages]
I20240503 13:57:06.614188 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:06.618078 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:06.624208 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:57:06.633802 13249 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 109 similar messages]
W20240503 13:57:06.634210 12987 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240503 13:57:06.634421 15558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240503 13:57:06.635303 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 13:57:06.639430 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 13:57:06.639628 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 13:57:06.641445 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:06.641479 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:06.641875 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:57:06.642513 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 13:57:06.643646 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:57:06.706589 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 13:57:06.715958 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 13:57:06.716583 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 13:57:06.737506 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 13:57:06.742712 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 13:57:06.765735 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 13:57:06.767009 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:06.772394 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 13:57:06.887518 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 13:57:06.901496 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 13:57:06.929634 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 13:57:06.983068 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 13:57:07.039180 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:07.043761 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:07.044588 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:07.047984 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:07.051576 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:07.072026 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:07.082624 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:57:07.355289 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:57:07.461091 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:07.471273 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:07.473237 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:57:07.479692 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 13:57:07.483244 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:07.484634 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:07.484810 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:07.488121 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:57:07.613869 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 13:57:07.689123 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 13:57:07.766316 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 13:57:07.770926 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 13:57:07.779320 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 13:57:07.840533 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:07.850814 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 13:57:07.886279 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:57:07.889715 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 13:57:07.891093 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:07.894449 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:07.896284 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:07.908136 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:07.909572 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:07.928876 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:57:07.948874 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 13:57:07.965387 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 13:57:08.005090 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 13:57:08.035032 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 13:57:08.081046 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 13:57:08.085425 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 13:57:08.089720 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 13:57:08.226423 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 13:57:08.235146 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 13:57:08.260938 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 13:57:08.277808 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 13:57:08.292047 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 13:57:08.317994 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 13:57:08.321961 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:08.322855 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:08.329524 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:08.334559 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:08.336956 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:57:08.338485 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:57:08.349892 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:08.371569 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:57:08.385284 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 13:57:08.392400 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 13:57:08.420957 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 13:57:08.459719 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 13:57:08.587322 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 13:57:08.623854 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 13:57:08.635361 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 13:57:08.708002 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 13:57:08.730733 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:08.735497 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:08.738487 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:08.749369 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:08.756882 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:57:08.777997 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 13:57:08.778492 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 13:57:08.790453 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:08.791204 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:57:08.805207 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:08.830654 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 13:57:08.859134 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 13:57:08.874846 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 13:57:08.877919 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 13:57:08.958123 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 13:57:08.991847 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 13:57:09.010598 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 13:57:09.035696 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 13:57:09.037227 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 13:57:09.045814 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 13:57:09.049212 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 13:57:09.054841 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 13:57:09.099562 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 13:57:09.159250 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 13:57:09.161716 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 13:57:09.163856 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:09.173867 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:09.174360 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:57:09.174589 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 13:57:09.180910 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:09.187768 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:09.201727 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:57:09.214342 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 13:57:09.240435 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 13:57:09.254134 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 13:57:09.254942 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 13:57:09.269157 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 13:57:09.273017 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 13:57:09.284756 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 13:57:09.304194 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 13:57:09.347293 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:09.349473 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 13:57:09.354430 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 13:57:09.428862 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 13:57:09.477401 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 13:57:09.525837 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 13:57:09.557435 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 13:57:09.577884 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:09.592252 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:57:09.594496 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 13:57:09.595633 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:09.600636 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:09.609130 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:09.612933 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:57:09.627347 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 13:57:09.637624 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:57:09.799897 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:09.947448 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 13:57:09.982561 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 13:57:09.994992 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:10.009380 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:10.010758 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:10.017765 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:10.022809 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:10.023550 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:10.023815 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:57:10.027806 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 13:57:10.064534 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 13:57:10.097577 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 13:57:10.131489 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 13:57:10.237058 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 13:57:10.312675 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 13:57:10.356349 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 13:57:10.366855 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 13:57:10.388993 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:10.413169 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 13:57:10.425055 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:10.426033 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:10.432129 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:10.436159 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:10.437050 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:10.457934 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:10.465788 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:57:10.544625 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 13:57:10.562870 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 13:57:10.570689 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 13:57:10.605754 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 13:57:10.671276 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 13:57:10.850672 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 13:57:10.853442 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 13:57:10.858750 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:57:10.859570 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 13:57:10.866462 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:10.869149 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:10.872184 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:10.877228 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:10.896662 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:57:10.897729 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 13:57:10.908687 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 13:57:10.926038 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 13:57:10.928581 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 13:57:10.930642 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 13:57:10.943313 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:11.009552 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 13:57:11.063580 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 13:57:11.079890 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 13:57:11.163519 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 13:57:11.179993 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 13:57:11.191313 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 13:57:11.223075 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 13:57:11.268959 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 13:57:11.301059 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:11.303428 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:11.306598 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:11.318358 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:11.318866 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:57:11.323792 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 13:57:11.325628 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 13:57:11.333640 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:11.345819 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:57:11.346078 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 13:57:11.347869 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 13:57:11.413157 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 13:57:11.427805 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 13:57:11.436476 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 13:57:11.440037 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 13:57:11.442142 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 13:57:11.474920 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 13:57:11.517355 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:11.531015 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 13:57:11.537918 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 13:57:11.548228 13118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 56 similar messages]
W20240503 13:57:11.553886 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 13:57:11.591395 15691 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 110 similar messages]
W20240503 13:57:11.592358 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 13:57:11.613229 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 13:57:11.614153 15426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 118 similar messages]
W20240503 13:57:11.621001 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 13:57:11.633303 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 13:57:11.640435 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 13:57:11.651595 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240503 13:57:11.652419 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 13:57:11.673197 13249 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 113 similar messages]
W20240503 13:57:11.684682 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 13:57:11.699539 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 13:57:11.735328 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:11.747849 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:11.748050 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:11.749141 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:11.754370 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:57:11.754848 15558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 92 similar messages]
I20240503 13:57:11.756400 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:57:11.773386 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 13:57:11.777730 12987 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 70 similar messages]
I20240503 13:57:11.798954 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:57:11.810895 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 13:57:11.817442 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 13:57:11.819928 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 13:57:11.911450 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:11.972303 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 13:57:12.015570 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 13:57:12.023288 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 13:57:12.102175 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 13:57:12.197243 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:12.201479 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:12.207442 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:12.215122 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:57:12.215173 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 13:57:12.234272 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:12.234395 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:57:12.254849 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 13:57:12.256701 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:57:12.325480 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 13:57:12.340018 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 13:57:12.408883 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 13:57:12.438134 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:12.463652 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 13:57:12.623581 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 13:57:12.644088 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:12.644248 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:12.646780 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:12.648905 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:12.663247 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:12.664772 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:57:12.677670 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 13:57:12.700245 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:57:12.731762 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 13:57:12.753346 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 13:57:12.774888 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 13:57:12.805629 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 13:57:12.954602 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:13.025403 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 13:57:13.077389 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:13.081060 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:13.085686 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:13.086521 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:13.087085 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:13.091001 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:57:13.099984 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 13:57:13.107587 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 13:57:13.117666 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:57:13.141893 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 13:57:13.152662 12896 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240503 13:57:13.153458 12896 catalog_manager.cc:4573] TS e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): CreateTablet RPC failed for tablet 2b362fed86c042868fc5b2fbaf83b369: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:13.191490 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 13:57:13.195261 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 13:57:13.258284 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 13:57:13.298132 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 13:57:13.348541 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:13.364676 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 13:57:13.374320 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 13:57:13.380587 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 13:57:13.418251 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 13:57:13.425154 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 13:57:13.496464 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 13:57:13.511359 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:57:13.520735 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 13:57:13.522570 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:13.523499 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:13.529889 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:13.532670 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:57:13.533025 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 13:57:13.547993 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:13.560994 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:57:13.666898 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 13:57:13.718358 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 13:57:13.744812 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 13:57:13.757324 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 13:57:13.790447 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 13:57:13.800047 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 13:57:13.804828 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:13.807571 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 13:57:13.845255 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 13:57:13.854279 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 13:57:13.857810 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 13:57:13.878540 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 13:57:13.899214 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 13:57:13.937222 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 13:57:13.937511 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 13:57:13.940133 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 13:57:13.941555 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 13:57:13.948292 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:13.953330 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:13.954535 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 13:57:13.954984 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 13:57:13.959300 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:13.960247 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:13.979297 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:13.979863 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:57:13.988133 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 13:57:14.064524 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 13:57:14.122674 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 13:57:14.150523 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 13:57:14.151234 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 13:57:14.158993 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 13:57:14.167405 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 13:57:14.171973 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 13:57:14.205613 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:14.219410 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 13:57:14.281136 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 13:57:14.361371 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 13:57:14.374784 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:14.374992 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:14.382041 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:14.385339 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:14.390010 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:14.393237 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:14.394491 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:57:14.404105 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 13:57:14.427244 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 13:57:14.430999 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 13:57:14.446300 12896 catalog_manager.cc:4573] TS e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): CreateTablet RPC failed for tablet 05ad17680502464ba4b195fa8b0c2dfc: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:14.449779 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 13:57:14.477519 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 13:57:14.550163 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 13:57:14.685241 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:14.724529 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 13:57:14.729203 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 13:57:14.742169 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 13:57:14.754628 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 13:57:14.754788 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 13:57:14.789628 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:14.790839 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:14.796829 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:14.798626 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:14.805214 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:14.805502 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:14.809718 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:57:14.838830 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 13:57:14.921236 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 13:57:14.966749 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 13:57:15.034682 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 13:57:15.127382 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 13:57:15.193691 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:57:15.259116 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:15.273898 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:15.275950 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:15.278412 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:15.300902 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:15.301651 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:15.301900 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:57:15.427493 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 13:57:15.505694 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 13:57:15.543817 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 13:57:15.617096 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 13:57:15.676077 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 13:57:15.691890 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 13:57:15.697110 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 13:57:15.702450 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:15.704375 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:15.709081 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:15.711133 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:15.716691 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:15.727893 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:15.732607 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:57:15.760458 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 13:57:15.777303 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 13:57:15.801095 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:15.826287 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 13:57:15.940868 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 13:57:15.941983 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 13:57:15.944025 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 13:57:15.991240 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 13:57:16.005784 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 13:57:16.021791 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 13:57:16.051635 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 13:57:16.059716 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 13:57:16.082497 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 13:57:16.103947 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 13:57:16.119706 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 13:57:16.121680 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:16.126098 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:16.129568 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:16.153916 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:16.155510 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:16.162186 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:16.170629 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:57:16.185802 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 13:57:16.194849 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 13:57:16.235584 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 13:57:16.253718 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 13:57:16.257645 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:16.261787 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 13:57:16.286469 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 13:57:16.322973 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 13:57:16.325346 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 13:57:16.347000 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 13:57:16.355808 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 13:57:16.360682 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 13:57:16.411772 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 13:57:16.546295 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 13:57:16.548974 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 13:57:16.563823 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 13:57:16.564309 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 13:57:16.568245 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:16.572811 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:16.576004 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:16.582561 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:57:16.586899 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 13:57:16.596390 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:16.596624 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:57:16.605043 13118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240503 13:57:16.637032 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 13:57:16.657232 15426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240503 13:57:16.657922 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 13:57:16.669951 15691 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 111 similar messages]
W20240503 13:57:16.670526 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 13:57:16.689628 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240503 13:57:16.690263 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:16.701035 13249 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 111 similar messages]
W20240503 13:57:16.701751 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 13:57:16.707351 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 13:57:16.761756 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 13:57:16.762246 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 13:57:16.771771 15558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240503 13:57:16.795372 12987 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240503 13:57:16.800566 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 13:57:16.838452 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 13:57:16.864183 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 13:57:16.891834 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 13:57:16.941736 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 13:57:16.946244 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 13:57:16.989687 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:16.995939 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:16.996783 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:17.002689 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:17.013496 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:17.014833 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:57:17.029412 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 13:57:17.047461 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 13:57:17.053248 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:57:17.082170 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 13:57:17.135392 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:17.144891 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 13:57:17.155640 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 13:57:17.195941 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 13:57:17.273011 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 13:57:17.316390 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 13:57:17.374332 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 13:57:17.411525 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:17.416445 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:17.418833 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:17.421095 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:17.430444 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:17.439807 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:17.444700 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:57:17.541594 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 13:57:17.550818 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 13:57:17.677377 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 13:57:17.709096 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 13:57:17.709826 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:57:17.841300 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:17.848829 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:17.855077 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:17.858180 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:17.869829 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:17.872977 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:17.886876 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:57:18.004942 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 13:57:18.011770 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 13:57:18.195015 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 13:57:18.257066 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 13:57:18.282647 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:18.283198 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:57:18.285249 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 13:57:18.286433 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 13:57:18.287883 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:18.293219 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:18.298543 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:18.300670 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:18.302299 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:57:18.325774 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 13:57:18.333951 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:18.375490 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 13:57:18.378191 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 13:57:18.408334 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 13:57:18.410398 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 13:57:18.464447 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 13:57:18.493657 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 13:57:18.551663 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 13:57:18.553498 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 13:57:18.614686 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 13:57:18.618566 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 13:57:18.642778 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 13:57:18.644451 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 13:57:18.669502 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 13:57:18.688540 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 13:57:18.704684 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:18.705417 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:18.706980 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:18.709841 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:18.720727 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:18.721567 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:57:18.730942 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 13:57:18.743726 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 13:57:18.756714 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:57:18.759479 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 13:57:18.766671 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 13:57:18.801115 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 13:57:18.812978 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:18.816687 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 13:57:18.828358 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 13:57:18.882838 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 13:57:18.897187 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 13:57:18.903431 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 13:57:18.938310 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 13:57:19.087463 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 13:57:19.111707 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:19.121620 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:19.129860 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:19.132305 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:57:19.134855 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 13:57:19.143399 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:57:19.145772 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 13:57:19.151280 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:19.176946 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:57:19.186882 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 13:57:19.194228 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 13:57:19.206748 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 13:57:19.208863 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 13:57:19.208993 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 13:57:19.238751 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 13:57:19.240358 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 13:57:19.249756 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 13:57:19.274130 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:19.323220 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 13:57:19.367734 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 13:57:19.397807 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 13:57:19.436187 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 13:57:19.524325 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 13:57:19.547716 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:19.551927 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:19.553683 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:19.555267 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:19.576287 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:19.577471 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:19.601089 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:57:19.685564 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 13:57:19.710345 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 13:57:19.712529 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 13:57:19.715986 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 13:57:19.721530 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:19.746068 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 13:57:19.827826 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 13:57:19.849259 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 13:57:19.866477 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 13:57:19.982864 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:19.985352 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:19.985383 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:19.987489 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:19.988022 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:20.003105 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:20.011212 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:57:20.052057 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 13:57:20.079092 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 13:57:20.086611 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 13:57:20.110633 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 13:57:20.122910 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:20.196277 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 13:57:20.348255 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 13:57:20.372243 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 13:57:20.395778 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:20.396664 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:20.400566 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:20.402160 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:20.405592 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:20.416364 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:20.416649 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:57:20.498153 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 13:57:20.500677 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 13:57:20.550369 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:20.556308 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 13:57:20.566220 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 13:57:20.656864 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 13:57:20.750958 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 13:57:20.815752 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:57:20.816020 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 13:57:20.818399 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:20.819589 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:20.826802 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:20.829377 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:20.835052 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:20.863505 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:57:20.870585 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 13:57:20.906823 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 13:57:20.923202 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 13:57:20.948124 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 13:57:21.042089 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 13:57:21.053813 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:21.097725 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 13:57:21.110080 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 13:57:21.143500 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 13:57:21.151017 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 13:57:21.160893 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 13:57:21.169829 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 13:57:21.217989 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 13:57:21.220137 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 13:57:21.220461 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 13:57:21.221980 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 13:57:21.255524 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:21.267397 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:21.267611 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:21.269769 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:21.272033 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:57:21.292568 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 13:57:21.293004 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:21.304917 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:57:21.316012 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 13:57:21.369741 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 13:57:21.422012 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 13:57:21.435976 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:21.436317 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 13:57:21.461936 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 13:57:21.498507 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 13:57:21.516861 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 13:57:21.522181 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 13:57:21.529421 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 13:57:21.534097 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 13:57:21.619374 13118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240503 13:57:21.620390 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 13:57:21.698992 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:57:21.701315 15426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 122 similar messages]
I20240503 13:57:21.702551 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:21.705386 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:21.709558 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:57:21.709888 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240503 13:57:21.709955 15691 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 108 similar messages]
W20240503 13:57:21.711215 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 13:57:21.715025 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:21.723403 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:21.724025 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:57:21.791006 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 13:57:21.795365 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 13:57:21.801609 13249 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 109 similar messages]
W20240503 13:57:21.802284 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 13:57:21.809343 15558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 87 similar messages]
W20240503 13:57:21.834926 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 13:57:21.860961 12987 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 66 similar messages]
W20240503 13:57:21.864418 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 13:57:21.864723 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 13:57:21.875342 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 13:57:21.916344 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 13:57:21.977767 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:21.982625 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 13:57:21.984333 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 13:57:22.102816 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:22.105165 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:22.109939 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:22.113118 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:22.126654 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:22.127233 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:57:22.142650 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 13:57:22.160827 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 13:57:22.165874 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:57:22.216264 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 13:57:22.222422 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 13:57:22.226378 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 13:57:22.245906 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 13:57:22.270300 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 13:57:22.285046 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 13:57:22.353961 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 13:57:22.414131 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 13:57:22.460841 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 13:57:22.469911 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 13:57:22.495405 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:57:22.527292 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:22.529062 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:22.529124 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:22.548666 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:22.552215 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:22.552656 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:22.579983 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:57:22.648255 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 13:57:22.699717 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 13:57:22.764639 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 13:57:22.769202 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 13:57:22.795313 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 13:57:22.901926 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:57:22.952605 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:22.953306 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:22.968037 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:22.968982 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:22.978147 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:22.980219 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:22.992671 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:57:23.032737 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 13:57:23.078444 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 13:57:23.168651 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 13:57:23.181350 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 13:57:23.229326 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 13:57:23.234140 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 13:57:23.247256 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 13:57:23.277753 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:57:23.378398 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:23.388180 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:23.389465 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:57:23.389477 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 13:57:23.390172 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:23.391252 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:23.394196 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:23.401753 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:57:23.416776 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 13:57:23.448396 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 13:57:23.457989 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 13:57:23.468636 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 13:57:23.492455 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 13:57:23.557564 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 13:57:23.569108 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 13:57:23.608283 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 13:57:23.677587 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 13:57:23.690451 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 13:57:23.705402 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 13:57:23.713153 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 13:57:23.713361 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 13:57:23.723703 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:23.767995 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 13:57:23.799208 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:23.809289 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:23.809384 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:23.817894 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:23.822445 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:23.825834 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:23.832170 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:57:23.871951 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 13:57:23.884351 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 13:57:23.891510 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 13:57:23.935773 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 13:57:23.967726 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 13:57:23.992827 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 13:57:24.013072 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 13:57:24.037281 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 13:57:24.094175 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 13:57:24.104025 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 13:57:24.159214 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 13:57:24.166710 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 13:57:24.190204 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 13:57:24.190629 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 13:57:24.207146 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 13:57:24.218860 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:24.228204 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:24.235116 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:24.246158 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:24.247520 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:57:24.248231 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:57:24.250559 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:24.259626 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:57:24.324355 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 13:57:24.384465 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 13:57:24.394021 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 13:57:24.414387 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 13:57:24.563323 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 13:57:24.573540 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 13:57:24.610615 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 13:57:24.614985 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 13:57:24.636888 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:57:24.651113 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:24.651418 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:24.655299 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:24.657510 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:24.658331 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:24.659648 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:57:24.669247 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 13:57:24.694231 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:57:24.761667 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 13:57:24.766844 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 13:57:24.797726 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 13:57:24.808871 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 13:57:24.852070 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 13:57:24.893361 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 13:57:24.949481 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 13:57:24.996034 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 13:57:24.998255 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 13:57:25.016479 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 13:57:25.050035 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 13:57:25.058151 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:25.061023 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:25.062783 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:25.070493 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:25.074119 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:57:25.080861 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 13:57:25.082233 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:25.096884 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:57:25.169410 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:25.196544 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 13:57:25.197502 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 13:57:25.321676 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 13:57:25.378373 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 13:57:25.420185 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 13:57:25.477402 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 13:57:25.477749 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:25.482358 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:25.483239 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:25.485266 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:25.487311 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:25.487977 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:25.504916 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:57:25.565351 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 13:57:25.569481 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 13:57:25.576272 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 13:57:25.600776 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 13:57:25.612133 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:25.669692 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 13:57:25.675956 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 13:57:25.698247 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 13:57:25.816006 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 13:57:25.887250 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:25.890205 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:25.894985 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:25.896378 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:25.896898 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:57:25.905341 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 13:57:25.909372 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 13:57:25.911980 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:25.923492 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:57:25.927321 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 13:57:25.938760 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 13:57:25.938920 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 13:57:25.971071 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 13:57:26.087488 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:26.094244 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 13:57:26.140551 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 13:57:26.263473 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 13:57:26.266346 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 13:57:26.278887 12896 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 6 similar messages]
W20240503 13:57:26.280467 12896 catalog_manager.cc:4573] TS e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): CreateTablet RPC failed for tablet 2b362fed86c042868fc5b2fbaf83b369: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:26.280601 12896 catalog_manager.cc:4605] Request timed out: CreateTablet RPC for tablet 2b362fed86c042868fc5b2fbaf83b369 on TS e47105a0f46e4511b7112e571688a4c7
I20240503 13:57:26.321123 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:26.324705 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:26.324996 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:26.327525 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:26.330037 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:57:26.337312 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 13:57:26.340970 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:57:26.360634 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 13:57:26.366348 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 13:57:26.366818 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:57:26.395072 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 13:57:26.402801 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 13:57:26.403343 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 13:57:26.414801 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 13:57:26.437546 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 13:57:26.438657 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 13:57:26.448791 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 13:57:26.480765 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 13:57:26.500190 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:26.512566 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 13:57:26.543390 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 13:57:26.557336 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 13:57:26.563251 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 13:57:26.590202 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 13:57:26.638564 13118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240503 13:57:26.711370 15426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240503 13:57:26.712384 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 13:57:26.740667 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:26.742444 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:26.743614 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 13:57:26.744127 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 13:57:26.752671 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:26.753163 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:26.762835 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:26.793855 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:57:26.808156 13249 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 109 similar messages]
W20240503 13:57:26.817895 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240503 13:57:26.869907 15691 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 113 similar messages]
W20240503 13:57:26.874084 12987 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240503 13:57:26.889988 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:26.905030 15558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240503 13:57:26.915323 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 13:57:26.929689 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 13:57:26.932113 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 13:57:27.010089 12896 catalog_manager.cc:4605] Request timed out: CreateTablet RPC for tablet ff21b7dd34b54f5f92a8300e81bc5b1c on TS e47105a0f46e4511b7112e571688a4c7
W20240503 13:57:27.024240 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 13:57:27.028021 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 13:57:27.077605 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 13:57:27.080307 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 13:57:27.102367 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 13:57:27.102784 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 13:57:27.163154 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:27.167944 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:27.177338 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:27.180788 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:27.182260 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:57:27.201447 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 13:57:27.207267 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:27.223418 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:57:27.237272 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 13:57:27.264860 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 13:57:27.299180 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 13:57:27.323827 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 13:57:27.325161 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 13:57:27.414937 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 13:57:27.474128 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 13:57:27.513327 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:27.516078 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 13:57:27.608238 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:27.610816 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:27.614656 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:27.617105 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:27.617350 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:27.631142 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:57:27.631445 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 13:57:27.641745 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 13:57:27.655654 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:57:27.688414 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 13:57:27.689483 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 13:57:27.721688 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 13:57:27.755129 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 13:57:27.789520 12896 catalog_manager.cc:4573] TS e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): CreateTablet RPC failed for tablet 05ad17680502464ba4b195fa8b0c2dfc: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:27.789611 12896 catalog_manager.cc:4605] Request timed out: CreateTablet RPC for tablet 05ad17680502464ba4b195fa8b0c2dfc on TS e47105a0f46e4511b7112e571688a4c7
W20240503 13:57:27.790189 12896 catalog_manager.cc:4605] Request timed out: CreateTablet RPC for tablet 9bef98a0c5e0411cb55141a93c62f5c0 on TS e47105a0f46e4511b7112e571688a4c7
W20240503 13:57:27.914069 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 13:57:27.994268 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 13:57:28.046746 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:28.047062 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:28.048179 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:28.049396 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:28.051206 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:28.056783 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:28.062394 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:57:28.064262 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 13:57:28.073573 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:28.124995 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 13:57:28.145975 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 13:57:28.159035 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 13:57:28.164810 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 13:57:28.198302 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 13:57:28.208998 12896 catalog_manager.cc:4605] Request timed out: CreateTablet RPC for tablet 7519f1091c6e4bdda68e8c7bafcff9f5 on TS e47105a0f46e4511b7112e571688a4c7
W20240503 13:57:28.225126 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 13:57:28.308416 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 13:57:28.311055 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 13:57:28.340831 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 13:57:28.367385 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 13:57:28.372159 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 13:57:28.445005 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:57:28.462114 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:28.464751 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:28.466995 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:28.477666 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:28.485324 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:28.503082 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:28.508988 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:57:28.574687 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 13:57:28.592761 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 13:57:28.604555 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 13:57:28.667217 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 13:57:28.706684 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 13:57:28.727463 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 13:57:28.734877 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 13:57:28.760339 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 13:57:28.855770 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 13:57:28.867942 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 13:57:28.887964 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:28.891920 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:28.892800 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:57:28.894026 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:28.894161 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 13:57:28.907011 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 13:57:28.913751 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:28.914227 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:28.925487 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:57:28.933674 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 13:57:28.944830 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:57:28.949273 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 13:57:28.950945 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 13:57:28.973613 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 13:57:29.005674 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 13:57:29.013849 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 13:57:29.033856 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 13:57:29.043901 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 13:57:29.056566 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 13:57:29.169057 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 13:57:29.184474 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 13:57:29.197189 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 13:57:29.218854 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 13:57:29.229616 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 13:57:29.337589 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:29.338320 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:29.340394 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:29.352113 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:29.363240 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:29.365481 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:29.377776 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:57:29.391502 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 13:57:29.425392 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 13:57:29.471094 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 13:57:29.484376 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:29.494290 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 13:57:29.595096 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 13:57:29.607991 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 13:57:29.610272 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 13:57:29.658195 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 13:57:29.667946 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 13:57:29.696763 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 13:57:29.753559 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:57:29.756697 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 13:57:29.758338 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:29.764365 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:29.766077 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:29.772697 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:57:29.776285 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 13:57:29.788440 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:29.815071 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:57:29.847506 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 13:57:29.920680 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 13:57:30.035617 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 13:57:30.043699 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 13:57:30.072785 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 13:57:30.078136 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:30.103904 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 13:57:30.201957 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:30.206408 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:30.210698 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:30.214478 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 13:57:30.219548 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 13:57:30.221662 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:57:30.238613 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 13:57:30.243801 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 13:57:30.244236 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:30.250186 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:57:30.282586 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 13:57:30.303875 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 13:57:30.341328 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 13:57:30.364745 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 13:57:30.408747 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 13:57:30.439239 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 13:57:30.462519 12896 catalog_manager.cc:4573] TS e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2427783ee0e348b98453818704ece5f8: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:30.563549 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 13:57:30.644346 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 13:57:30.655845 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:30.659808 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:30.663091 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:30.674746 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:57:30.678260 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:57:30.680826 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:30.689347 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:57:30.691938 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 13:57:30.701376 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:57:30.709213 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 13:57:30.720983 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 13:57:30.816895 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 13:57:30.834576 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 13:57:30.889068 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 13:57:30.893843 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 13:57:30.949980 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 13:57:30.957739 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 13:57:31.047324 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 13:57:31.058782 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 13:57:31.072152 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:31.072964 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:31.076862 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:31.077446 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:31.079260 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:31.087641 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:31.098591 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 13:57:31.151988 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 13:57:31.153307 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:31.232934 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 13:57:31.240760 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 13:57:31.275797 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 13:57:31.288981 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 13:57:31.301241 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 13:57:31.361595 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 13:57:31.369009 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 13:57:31.374800 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 13:57:31.429093 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 13:57:31.450291 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 13:57:31.476784 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:31.486210 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:31.493639 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:31.494509 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:31.495261 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:57:31.495545 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 13:57:31.504405 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:31.525355 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:57:31.528597 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 13:57:31.536464 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:31.614913 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 13:57:31.627528 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 13:57:31.688391 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 13:57:31.700413 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 13:57:31.706514 13118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240503 13:57:31.712819 15426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 122 similar messages]
W20240503 13:57:31.713521 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 13:57:31.714046 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 13:57:31.723557 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 13:57:31.752892 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 13:57:31.759644 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 13:57:31.787029 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 13:57:31.835367 13249 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 110 similar messages]
W20240503 13:57:31.839126 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 13:57:31.852569 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240503 13:57:31.853806 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 13:57:31.863648 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 13:57:31.867846 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 13:57:31.896551 12987 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240503 13:57:31.906150 15691 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 111 similar messages]
W20240503 13:57:31.908761 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 13:57:31.936367 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 13:57:31.950917 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:31.953040 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:31.954569 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:31.957008 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:31.986717 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:57:31.991230 15558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 93 similar messages]
I20240503 13:57:31.996085 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:31.999539 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:57:32.040261 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:32.173226 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 13:57:32.247963 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 13:57:32.343104 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 13:57:32.344108 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 13:57:32.374879 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:32.375365 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:32.383638 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:32.384698 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:32.386883 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:32.407450 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:32.428375 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:57:32.454056 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 13:57:32.468026 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 13:57:32.496956 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 13:57:32.502496 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 13:57:32.517964 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 13:57:32.545856 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:32.552387 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 13:57:32.562876 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 13:57:32.629151 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 13:57:32.697798 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 13:57:32.705909 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 13:57:32.800439 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 13:57:32.811939 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 13:57:32.814030 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 13:57:32.814190 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 13:57:32.814162 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:32.814177 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:32.815557 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:32.830022 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:57:32.836006 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 13:57:32.836301 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:32.839974 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:57:32.941684 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 13:57:32.944991 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 13:57:32.976424 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 13:57:33.004153 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:33.034626 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 13:57:33.130816 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 13:57:33.218438 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 13:57:33.247598 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:33.254581 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:33.258618 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:33.258780 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:33.264292 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:33.284924 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:33.285567 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:57:33.310461 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 13:57:33.318889 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 13:57:33.325711 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 13:57:33.363317 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 13:57:33.392591 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 13:57:33.446761 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 13:57:33.457350 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:33.579707 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 13:57:33.626801 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 13:57:33.627364 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 13:57:33.641345 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 13:57:33.651676 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 13:57:33.693425 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:33.694512 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 13:57:33.697458 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 13:57:33.711944 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:33.712261 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:33.718119 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:33.719022 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:33.725764 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:57:33.734601 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 13:57:33.754138 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 13:57:33.754841 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 13:57:33.795310 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 13:57:33.802894 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 13:57:33.876760 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:33.894023 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 13:57:33.931169 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 13:57:33.968433 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 13:57:34.005834 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 13:57:34.024739 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 13:57:34.052930 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 13:57:34.098111 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 13:57:34.113422 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:34.118975 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:34.122854 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:34.123824 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:34.125615 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:34.134655 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:34.144404 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:57:34.167559 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 13:57:34.170598 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 13:57:34.204026 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 13:57:34.214457 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 13:57:34.221407 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 13:57:34.243466 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 13:57:34.248659 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 13:57:34.291860 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 13:57:34.349889 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 13:57:34.381319 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:34.472448 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 13:57:34.496796 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 13:57:34.529024 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:34.533474 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:34.534646 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:34.540253 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:34.546334 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:34.547132 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:57:34.548862 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 13:57:34.550539 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:57:34.566481 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 13:57:34.605769 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 13:57:34.614418 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 13:57:34.636262 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 13:57:34.699630 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 13:57:34.855235 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:34.948839 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 13:57:34.956097 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:34.963757 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:57:34.971974 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 13:57:34.972105 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:34.973250 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:57:34.975589 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 13:57:34.975622 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:34.983590 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:57:34.992781 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 13:57:34.995054 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:57:35.032894 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 13:57:35.034706 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 13:57:35.233605 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 13:57:35.251703 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 13:57:35.272418 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:35.281524 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 13:57:35.305282 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 13:57:35.311040 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 13:57:35.320766 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 13:57:35.344144 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 13:57:35.380460 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:35.386644 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:35.390033 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:35.397760 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:35.412786 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:35.413281 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:35.413820 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 13:57:35.425961 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 13:57:35.453310 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 13:57:35.486860 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 13:57:35.534579 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 13:57:35.580318 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 13:57:35.592407 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 13:57:35.611186 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 13:57:35.649149 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 13:57:35.773566 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 13:57:35.780200 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:57:35.808051 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:35.808915 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:35.812247 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:35.814391 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:35.823045 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:57:35.825814 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 13:57:35.826962 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:35.831791 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:57:35.965296 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 13:57:35.972710 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 13:57:35.984145 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 13:57:35.992878 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 13:57:36.008072 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 13:57:36.044747 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 13:57:36.153751 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 13:57:36.182416 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 13:57:36.183593 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 13:57:36.239483 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:36.247628 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:36.256033 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:36.261296 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:57:36.262779 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 13:57:36.267246 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:36.267614 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:36.269114 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:57:36.284395 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 13:57:36.305682 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 13:57:36.321126 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 13:57:36.338989 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:36.390447 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 13:57:36.399145 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 13:57:36.421638 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 13:57:36.455169 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 13:57:36.469961 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 13:57:36.528815 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 13:57:36.542775 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 13:57:36.583226 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 13:57:36.605188 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 13:57:36.634567 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 13:57:36.659282 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 13:57:36.671713 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 13:57:36.674268 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:36.676913 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:36.686371 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:36.692848 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:36.693992 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:36.694123 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:57:36.696877 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 13:57:36.707729 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:57:36.721050 15426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240503 13:57:36.723294 13118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 57 similar messages]
W20240503 13:57:36.723274 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 13:57:36.731637 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:36.798193 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 13:57:36.798650 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 13:57:36.809803 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 13:57:36.920816 15691 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 109 similar messages]
W20240503 13:57:36.924618 13249 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 109 similar messages]
W20240503 13:57:36.952579 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 13:57:36.961989 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240503 13:57:36.962814 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 13:57:36.987905 12987 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240503 13:57:36.988718 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 13:57:37.005332 15558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240503 13:57:37.022410 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 13:57:37.065531 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 13:57:37.089344 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:37.091899 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:37.100265 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:37.100682 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:37.107489 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:37.110353 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:37.117427 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:57:37.128312 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 13:57:37.164904 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 13:57:37.188771 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 13:57:37.200273 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 13:57:37.240244 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:37.275756 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 13:57:37.353850 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 13:57:37.404579 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 13:57:37.490175 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 13:57:37.499920 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:37.501852 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:37.510406 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:37.516779 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:57:37.517688 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 13:57:37.517920 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:37.520612 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:57:37.536886 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 13:57:37.550825 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:57:37.593457 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 13:57:37.609872 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 13:57:37.648137 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 13:57:37.727434 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 13:57:37.736171 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 13:57:37.830579 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 13:57:37.842131 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 13:57:37.843230 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 13:57:37.848174 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:37.870373 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 13:57:37.924193 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:37.930219 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:37.931030 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:37.932332 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:37.942742 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:37.948201 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:37.949193 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:57:38.006124 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 13:57:38.030623 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 13:57:38.154234 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 13:57:38.171164 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 13:57:38.183171 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 13:57:38.206558 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 13:57:38.226279 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:38.228399 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 13:57:38.306066 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 13:57:38.353670 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:38.359146 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:38.359757 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:38.372575 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:38.372776 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:38.384518 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:38.407337 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:57:38.486950 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 13:57:38.530067 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 13:57:38.539362 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 13:57:38.552369 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 13:57:38.600739 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 13:57:38.621572 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 13:57:38.652789 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:38.660822 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 13:57:38.723417 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 13:57:38.724110 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 13:57:38.747504 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 13:57:38.758710 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 13:57:38.776986 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:38.787459 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:38.789233 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:38.791441 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:38.796435 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:38.798063 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:38.800568 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:57:38.851830 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 13:57:38.859663 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 13:57:38.894454 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 13:57:38.944705 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 13:57:38.958678 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 13:57:38.983700 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 13:57:38.999431 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 13:57:39.001297 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 13:57:39.026679 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 13:57:39.041556 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 13:57:39.045416 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 13:57:39.132347 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 13:57:39.157917 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 13:57:39.195047 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:39.197135 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:39.198393 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:39.198845 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:39.206713 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:39.235373 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:57:39.239495 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 13:57:39.243220 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:39.243785 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 13:57:39.246536 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:57:39.249773 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 13:57:39.253669 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 13:57:39.289091 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 13:57:39.350142 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 13:57:39.360144 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 13:57:39.419454 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 13:57:39.440909 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 13:57:39.469882 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 13:57:39.472640 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 13:57:39.479848 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 13:57:39.563591 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 13:57:39.571359 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 13:57:39.633210 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:39.637516 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:39.637727 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:39.638352 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:39.641566 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:39.644742 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 13:57:39.659188 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:57:39.674842 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:57:39.696318 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 13:57:39.728123 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 13:57:39.767617 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 13:57:39.867985 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 13:57:39.888154 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 13:57:39.897007 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 13:57:40.036397 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 13:57:40.049095 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:40.049600 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:40.052032 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:40.056535 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:40.059159 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:40.064198 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:40.064762 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:57:40.068614 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 13:57:40.096961 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 13:57:40.121637 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:40.138510 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 13:57:40.149710 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 13:57:40.186203 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 13:57:40.232748 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 13:57:40.260967 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 13:57:40.272866 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 13:57:40.458488 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:40.468700 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:40.469806 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:40.471027 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:40.473137 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:57:40.482161 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 13:57:40.484700 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:40.507740 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:57:40.510529 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 13:57:40.534996 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 13:57:40.541297 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 13:57:40.627836 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 13:57:40.639983 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:40.645218 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 13:57:40.650938 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 13:57:40.687435 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 13:57:40.785284 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 13:57:40.841984 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 13:57:40.876333 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:40.881297 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:40.882936 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:40.894809 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:57:40.901072 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 13:57:40.901759 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 13:57:40.903677 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:40.905295 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:40.912657 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:57:40.928570 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 13:57:40.989507 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 13:57:41.081954 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 13:57:41.087815 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 13:57:41.140529 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 13:57:41.146436 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 13:57:41.155918 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:41.291597 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 13:57:41.321308 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:41.323367 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:41.326954 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:57:41.327468 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 13:57:41.339843 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 13:57:41.341218 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:41.341289 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:57:41.343708 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 13:57:41.348009 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:57:41.356335 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 13:57:41.375437 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 13:57:41.380569 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 13:57:41.422463 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 13:57:41.445080 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 13:57:41.445451 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 13:57:41.450261 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 13:57:41.488906 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 13:57:41.518317 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 13:57:41.521760 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 13:57:41.538853 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 13:57:41.538902 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 13:57:41.565066 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 13:57:41.645709 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 13:57:41.683965 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 13:57:41.692023 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:57:41.741115 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:57:41.742841 15426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 120 similar messages]
I20240503 13:57:41.742878 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:41.745942 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:41.751308 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:41.754277 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:41.775990 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:57:41.786407 13118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240503 13:57:41.790477 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 13:57:41.799407 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:57:41.821147 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 13:57:41.822258 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 13:57:41.853554 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 13:57:41.869904 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 13:57:41.892230 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 13:57:41.940645 13249 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 113 similar messages]
W20240503 13:57:41.982928 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240503 13:57:41.983534 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 13:57:41.988058 12987 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240503 13:57:41.988654 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 13:57:42.012598 15691 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 111 similar messages]
W20240503 13:57:42.040380 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 13:57:42.061228 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 13:57:42.087311 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 13:57:42.102123 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:42.110751 15558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 91 similar messages]
I20240503 13:57:42.174829 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:42.175014 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:42.179200 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:42.184927 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:42.213043 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:42.214246 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:42.214689 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:57:42.253407 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 13:57:42.370826 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 13:57:42.410210 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 13:57:42.446733 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 13:57:42.460163 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 13:57:42.474627 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 13:57:42.485146 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 13:57:42.485524 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 13:57:42.599815 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 13:57:42.600342 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:42.606575 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:42.612639 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:42.614432 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:42.614585 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:42.616115 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:42.629132 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:57:42.662684 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 13:57:42.707713 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:42.720163 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 13:57:42.741153 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 13:57:42.752548 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 13:57:42.810254 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 13:57:42.870785 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 13:57:42.934602 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 13:57:42.956425 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 13:57:43.024174 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:43.029618 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:43.035380 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:43.035465 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:43.038520 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:43.039885 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:43.047859 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:57:43.059484 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 13:57:43.075675 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 13:57:43.097435 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 13:57:43.126789 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 13:57:43.131268 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 13:57:43.185810 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 13:57:43.190335 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:43.296260 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 13:57:43.330439 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 13:57:43.369345 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 13:57:43.413640 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 13:57:43.432137 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 13:57:43.436204 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:43.438331 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:43.441923 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:43.452288 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:43.453434 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:43.474893 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:43.485615 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:57:43.526269 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 13:57:43.531358 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 13:57:43.600553 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 13:57:43.654834 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 13:57:43.670786 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 13:57:43.678568 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 13:57:43.686139 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 13:57:43.710785 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:43.757707 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 13:57:43.777334 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 13:57:43.855010 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:43.859441 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:43.860591 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:43.865869 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:57:43.870679 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 13:57:43.874619 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:43.876745 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:57:43.894446 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 13:57:43.901331 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:57:43.903947 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 13:57:43.939666 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 13:57:43.955202 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 13:57:44.010993 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 13:57:44.027160 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 13:57:44.048324 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 13:57:44.079879 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 13:57:44.089591 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:44.112985 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 13:57:44.125170 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 13:57:44.129722 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 13:57:44.216001 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 13:57:44.228525 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 13:57:44.233578 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 13:57:44.275000 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 13:57:44.288686 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 13:57:44.290887 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 13:57:44.298370 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:44.304468 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:44.304658 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:44.305109 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:44.307024 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:44.310701 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:57:44.311031 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 13:57:44.404726 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 13:57:44.414808 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 13:57:44.559168 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 13:57:44.590243 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 13:57:44.626101 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 13:57:44.647234 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:44.680730 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 13:57:44.693950 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:44.695187 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:44.695930 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:44.707568 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:44.711560 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:44.723799 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:44.730098 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:57:44.736184 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 13:57:44.789903 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 13:57:44.797338 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 13:57:44.812764 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 13:57:44.818457 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 13:57:44.922257 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 13:57:44.943096 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 13:57:44.958359 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 13:57:45.020630 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 13:57:45.026079 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 13:57:45.056035 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 13:57:45.107935 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:45.108860 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:45.109072 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:45.117074 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:45.124517 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:57:45.124636 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:57:45.131980 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:45.151616 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:57:45.165197 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 13:57:45.208962 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 13:57:45.218914 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 13:57:45.220436 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 13:57:45.333281 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 13:57:45.356637 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 13:57:45.506105 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 13:57:45.547056 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 13:57:45.549072 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 13:57:45.550132 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:45.565371 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:57:45.571269 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 13:57:45.579941 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:57:45.582975 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 13:57:45.584513 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:45.592705 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:45.635447 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:57:45.678705 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 13:57:45.688259 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 13:57:45.720561 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 13:57:45.728272 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:45.771417 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 13:57:45.777532 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 13:57:45.820569 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 13:57:45.913111 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 13:57:46.010901 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:46.014199 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:46.023391 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:46.025112 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:46.026008 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:46.032649 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:46.045156 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 13:57:46.065949 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 13:57:46.085314 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 13:57:46.122953 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 13:57:46.133366 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 13:57:46.169059 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 13:57:46.204705 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 13:57:46.222183 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 13:57:46.237155 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 13:57:46.256304 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 13:57:46.261689 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:46.287273 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 13:57:46.342818 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 13:57:46.385906 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 13:57:46.427925 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 13:57:46.428036 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 13:57:46.430820 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:46.438899 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:46.439038 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:46.452659 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:57:46.458935 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 13:57:46.459790 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:46.469182 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:57:46.478900 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 13:57:46.485810 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 13:57:46.485939 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 13:57:46.504559 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 13:57:46.550719 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 13:57:46.553544 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 13:57:46.572671 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 13:57:46.636425 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 13:57:46.639961 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 13:57:46.676170 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 13:57:46.729071 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 13:57:46.732828 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 13:57:46.831926 13118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240503 13:57:46.842693 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:57:46.844949 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 13:57:46.848691 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:46.851176 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:57:46.855333 15426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 124 similar messages]
I20240503 13:57:46.866742 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:46.873675 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:46.881070 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:57:46.882753 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:57:46.886544 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:57:46.899225 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 13:57:46.905261 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 13:57:46.930184 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 13:57:46.934844 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 13:57:47.005486 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240503 13:57:47.006204 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 13:57:47.018304 12987 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 67 similar messages]
W20240503 13:57:47.018921 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 13:57:47.065587 13249 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 109 similar messages]
W20240503 13:57:47.069518 15691 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 105 similar messages]
W20240503 13:57:47.120574 15558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240503 13:57:47.132401 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 13:57:47.153872 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 13:57:47.171905 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 13:57:47.277946 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 13:57:47.290940 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 13:57:47.291435 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:47.291709 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:47.298218 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:47.298660 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:47.304008 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:47.305316 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:47.310477 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:57:47.320034 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 13:57:47.353864 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:47.372113 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 13:57:47.416664 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 13:57:47.538466 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 13:57:47.558701 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 13:57:47.612327 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 13:57:47.642607 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 13:57:47.703603 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:47.709335 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:47.713658 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:47.713958 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:47.718246 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:47.728713 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:47.764784 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:57:47.806388 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 13:57:47.847155 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 13:57:47.849390 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:47.863866 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 13:57:47.875373 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 13:57:47.977378 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 13:57:48.009174 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 13:57:48.012893 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 13:57:48.026126 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 13:57:48.052769 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 13:57:48.054548 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 13:57:48.081743 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 13:57:48.129330 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:48.132222 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:48.142570 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:48.150440 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:48.175057 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:48.175273 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:48.175590 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:57:48.200757 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 13:57:48.258409 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 13:57:48.323951 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 13:57:48.393966 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 13:57:48.468112 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 13:57:48.474993 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:48.524389 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 13:57:48.542050 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 13:57:48.572036 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:48.572211 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:48.572314 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:48.578832 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:48.579419 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:48.591413 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:57:48.598094 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 13:57:48.600065 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:57:48.624501 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 13:57:48.625370 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 13:57:48.628111 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 13:57:48.695746 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 13:57:48.716230 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 13:57:48.723163 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 13:57:48.763355 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 13:57:48.790656 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 13:57:48.801265 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 13:57:48.818352 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 13:57:48.882004 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 13:57:48.898628 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:48.939704 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 13:57:48.985471 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:48.989627 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:48.996912 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:48.999708 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:49.004667 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:49.006677 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:49.009802 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:57:49.044987 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 13:57:49.065371 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 13:57:49.077780 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 13:57:49.111308 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 13:57:49.173555 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 13:57:49.179288 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 13:57:49.197715 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 13:57:49.261783 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 13:57:49.266016 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 13:57:49.278560 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 13:57:49.284354 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 13:57:49.289227 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 13:57:49.301761 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 13:57:49.349182 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 13:57:49.364032 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 13:57:49.389617 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:57:49.404117 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:49.407123 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 13:57:49.408975 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 13:57:49.412189 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:49.412710 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:49.413170 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:49.414172 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:49.421826 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:57:49.457505 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 13:57:49.488016 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 13:57:49.496357 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 13:57:49.515959 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 13:57:49.521528 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 13:57:49.546140 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 13:57:49.563751 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 13:57:49.661759 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 13:57:49.812357 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 13:57:49.813894 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 13:57:49.815760 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:49.820829 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:49.824023 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:49.827179 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:49.833293 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:49.833732 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:57:49.886399 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 13:57:49.889654 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:49.915896 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 13:57:49.933344 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 13:57:49.940974 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 13:57:49.961481 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 13:57:50.020983 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 13:57:50.240134 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:50.246966 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:50.257612 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:50.261873 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:50.264675 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:50.276645 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:57:50.279670 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 13:57:50.287029 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 13:57:50.288539 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:57:50.338712 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:50.339784 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 13:57:50.385629 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 13:57:50.407967 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 13:57:50.481236 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 13:57:50.581068 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 13:57:50.593840 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 13:57:50.626992 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 13:57:50.629163 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 13:57:50.656044 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 13:57:50.673022 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:50.679142 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:50.682001 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:50.684163 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:50.684345 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:50.687093 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:50.691843 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:57:50.785153 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 13:57:50.826198 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:50.854207 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 13:57:50.914824 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 13:57:50.965092 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 13:57:50.969349 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 13:57:51.004777 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 13:57:51.067310 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 13:57:51.082937 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:51.091322 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:51.092212 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:51.097476 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:51.101322 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:51.102839 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:51.103057 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:57:51.149008 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 13:57:51.155448 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 13:57:51.155815 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 13:57:51.205479 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 13:57:51.217640 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 13:57:51.248292 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 13:57:51.297741 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 13:57:51.307938 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 13:57:51.310066 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 13:57:51.330233 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 13:57:51.343570 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 13:57:51.347723 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 13:57:51.366583 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:51.500864 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 13:57:51.501354 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:51.502094 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:57:51.502979 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 13:57:51.513878 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:51.516655 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:51.518286 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:57:51.521090 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 13:57:51.527581 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:57:51.537636 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 13:57:51.545024 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 13:57:51.553012 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:57:51.580919 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 13:57:51.629671 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 13:57:51.712936 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 13:57:51.714426 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 13:57:51.725163 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 13:57:51.733438 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 13:57:51.741950 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 13:57:51.762408 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 13:57:51.776844 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 13:57:51.855736 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 13:57:51.916635 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 13:57:51.922901 15426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240503 13:57:51.923600 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 13:57:51.930747 13118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240503 13:57:51.932117 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:51.938968 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:51.942478 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:51.944242 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:51.957625 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:51.968664 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:57:51.971288 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:51.972651 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 13:57:51.979482 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:57:51.979874 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 13:57:52.002351 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 13:57:52.053460 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240503 13:57:52.053880 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 13:57:52.054100 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 13:57:52.080840 12987 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240503 13:57:52.082484 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 13:57:52.089918 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 13:57:52.128079 13249 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 115 similar messages]
W20240503 13:57:52.129868 15558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240503 13:57:52.130546 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 13:57:52.132244 15691 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 114 similar messages]
W20240503 13:57:52.164516 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 13:57:52.182262 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 13:57:52.255637 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 13:57:52.308779 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 13:57:52.345649 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:57:52.348251 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:52.354068 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:52.362282 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:52.363070 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:52.375088 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:52.375432 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:52.376461 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:57:52.395795 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 13:57:52.402832 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 13:57:52.520251 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 13:57:52.625563 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 13:57:52.672657 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 13:57:52.727682 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:57:52.758090 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:52.769520 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:52.774538 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:52.777247 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:52.780076 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:52.785106 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:57:52.785923 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 13:57:52.805079 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 13:57:52.823520 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:57:52.827311 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 13:57:52.884239 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 13:57:52.925091 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 13:57:53.115715 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 13:57:53.121093 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 13:57:53.157189 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 13:57:53.199143 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:53.199774 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:53.202838 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:53.204102 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:53.212894 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:53.219744 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:57:53.220412 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 13:57:53.224637 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 13:57:53.226089 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 13:57:53.257961 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:57:53.326237 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:53.432745 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 13:57:53.452273 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 13:57:53.464390 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 13:57:53.508397 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 13:57:53.548686 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 13:57:53.554411 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 13:57:53.563162 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 13:57:53.614766 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 13:57:53.624454 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 13:57:53.638057 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:53.640422 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:53.644778 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:53.645581 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:53.645874 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:57:53.647082 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 13:57:53.652267 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:53.658473 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:57:53.688975 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 13:57:53.746131 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 13:57:53.748757 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 13:57:53.792187 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 13:57:53.802579 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:53.881598 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 13:57:53.924973 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 13:57:53.943692 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 13:57:54.032682 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 13:57:54.056286 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 13:57:54.069433 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:54.070206 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:54.071238 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:54.071553 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:54.095439 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:54.100736 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:54.106398 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:57:54.117650 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 13:57:54.118937 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 13:57:54.123366 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 13:57:54.132284 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 13:57:54.137969 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 13:57:54.150168 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 13:57:54.160538 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 13:57:54.176046 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 13:57:54.217576 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 13:57:54.257194 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 13:57:54.259402 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 13:57:54.262506 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 13:57:54.320405 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 13:57:54.329975 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:54.334997 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 13:57:54.378873 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 13:57:54.408787 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 13:57:54.500167 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:54.503074 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:54.506618 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:54.509753 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:54.510641 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 13:57:54.512545 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 13:57:54.512683 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:54.537278 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:57:54.581887 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 13:57:54.610869 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 13:57:54.646517 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 13:57:54.706866 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 13:57:54.752053 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 13:57:54.753947 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 13:57:54.807611 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 13:57:54.812175 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:54.823138 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 13:57:54.825085 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 13:57:54.914358 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:54.917235 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:54.920421 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:54.926111 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:54.928681 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:54.929328 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 13:57:54.936942 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 13:57:54.943282 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 13:57:54.964532 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:57:55.016691 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 13:57:55.141129 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 13:57:55.149859 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 13:57:55.194362 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 13:57:55.226166 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 13:57:55.262491 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 13:57:55.358878 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:55.361440 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:55.364604 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:55.369864 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:55.382373 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:57:55.384711 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:57:55.385427 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:55.406934 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:57:55.413689 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 13:57:55.427189 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 13:57:55.461649 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 13:57:55.538602 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 13:57:55.593098 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 13:57:55.595422 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 13:57:55.699308 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 13:57:55.715066 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 13:57:55.783649 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 13:57:55.791638 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:55.791657 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:55.802031 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:55.808156 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:55.812677 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:57:55.817121 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 13:57:55.822561 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 13:57:55.832438 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:55.836663 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:57:55.938988 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:56.027508 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 13:57:56.028357 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 13:57:56.047015 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 13:57:56.050791 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 13:57:56.080726 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 13:57:56.098155 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 13:57:56.104779 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 13:57:56.127832 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 13:57:56.207347 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 13:57:56.230854 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:56.235805 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:56.242446 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:56.243230 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:57:56.249032 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 13:57:56.251821 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:57:56.266490 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 13:57:56.268970 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 13:57:56.269095 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:56.272123 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:57:56.307718 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 13:57:56.308080 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 13:57:56.367070 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 13:57:56.380355 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 13:57:56.506852 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 13:57:56.519364 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 13:57:56.550189 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:56.566175 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 13:57:56.581234 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 13:57:56.602325 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 13:57:56.617453 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 13:57:56.652815 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 13:57:56.657806 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 13:57:56.663791 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:56.664098 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:56.670969 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:56.674867 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:56.695412 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:56.698789 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:57:56.702764 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 13:57:56.707178 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 13:57:56.711369 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 13:57:56.713138 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 13:57:56.738942 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 13:57:56.760609 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 13:57:56.810659 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 13:57:56.849228 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 13:57:56.875990 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 13:57:56.901939 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 13:57:56.956707 13118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240503 13:57:56.988447 15426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240503 13:57:57.052314 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 13:57:57.061712 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 13:57:57.072844 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:57.072929 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:57.074164 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 13:57:57.080287 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 70 similar messages]
I20240503 13:57:57.088109 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:57.092062 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:57.103964 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:57:57.107820 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 13:57:57.120000 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 13:57:57.123014 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 13:57:57.126770 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:57:57.130698 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:57:57.140590 15558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240503 13:57:57.159258 15691 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 111 similar messages]
W20240503 13:57:57.160089 13249 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 107 similar messages]
W20240503 13:57:57.161020 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 13:57:57.187345 12987 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 67 similar messages]
W20240503 13:57:57.188095 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 13:57:57.211627 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 13:57:57.216437 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 13:57:57.298375 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 13:57:57.335789 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 13:57:57.345484 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 13:57:57.415270 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 13:57:57.493376 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:57.500372 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:57.502727 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:57.506314 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:57.520025 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:57.525254 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:57.529398 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:57:57.537292 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 13:57:57.673516 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 13:57:57.673993 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 13:57:57.733863 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:57.782646 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 13:57:57.889477 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 13:57:57.939074 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:57.939534 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:57.950620 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:57.952083 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:57.959126 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:57.970700 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:57:57.975018 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 13:57:57.985703 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:57:58.045857 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 13:57:58.054401 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 13:57:58.072602 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 13:57:58.077877 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 13:57:58.081422 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 13:57:58.157903 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 13:57:58.218920 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:58.246981 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 13:57:58.319131 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 13:57:58.326288 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 13:57:58.360013 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:58.369736 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:58.372916 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:58.377487 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:58.381325 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:58.381532 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:58.385993 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:57:58.386988 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 13:57:58.511332 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 13:57:58.565316 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 13:57:58.594285 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 13:57:58.634240 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:58.634588 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 13:57:58.638952 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 13:57:58.671195 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 13:57:58.683223 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 13:57:58.683602 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 13:57:58.762279 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 13:57:58.781292 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 13:57:58.783521 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 13:57:58.798414 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:58.799026 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:58.801875 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:58.805444 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:58.809188 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:58.812672 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:57:58.823179 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 13:57:58.830703 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 13:57:58.845288 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 13:57:58.852049 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:57:58.869335 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 13:57:58.913672 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 13:57:58.951325 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 13:57:59.009275 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 13:57:59.028070 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 13:57:59.073563 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:59.085374 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 13:57:59.093720 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 13:57:59.098906 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 13:57:59.108836 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 13:57:59.186189 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 13:57:59.187621 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 13:57:59.248767 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:59.253548 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:59.257652 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:57:59.259550 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:59.268643 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:59.278993 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:57:59.287106 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 13:57:59.297015 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:57:59.341558 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 13:57:59.347503 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 13:57:59.353420 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 13:57:59.355429 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 13:57:59.368851 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 13:57:59.382000 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 13:57:59.408030 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 13:57:59.419034 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 13:57:59.515376 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 13:57:59.516866 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 13:57:59.530629 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 13:57:59.542380 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:57:59.567763 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 13:57:59.672283 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:57:59.680316 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:57:59.684300 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:57:59.688954 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 13:57:59.690448 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:57:59.691319 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:57:59.712554 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:57:59.717311 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:57:59.747704 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 13:57:59.786571 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 13:57:59.841316 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 13:57:59.842979 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 13:57:59.854149 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 13:57:59.863171 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 13:57:59.961149 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 13:58:00.012815 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 13:58:00.067790 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:58:00.088666 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:00.089768 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:00.095227 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:00.098732 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:00.105155 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:00.106388 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:00.128680 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:58:00.153446 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 13:58:00.288806 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 13:58:00.292281 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 13:58:00.425380 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 13:58:00.477479 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 13:58:00.508056 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:00.508339 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:00.512260 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:00.519250 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:00.525271 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:00.528317 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:58:00.540081 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 13:58:00.545248 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:58:00.563722 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 13:58:00.571214 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:00.576691 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 13:58:00.596294 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 13:58:00.623625 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 13:58:00.636627 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 13:58:00.671908 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 13:58:00.676366 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 13:58:00.873503 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 13:58:00.915949 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 13:58:00.919411 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 13:58:00.923422 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:00.927101 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:00.932188 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:00.936851 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:58:00.937105 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 13:58:00.942265 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:00.981016 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:58:01.026214 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 13:58:01.097380 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:01.109831 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 13:58:01.134796 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 13:58:01.180960 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 13:58:01.183344 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 13:58:01.185484 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 13:58:01.256549 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 13:58:01.271550 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 13:58:01.277536 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 13:58:01.295071 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 13:58:01.318696 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 13:58:01.327701 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 13:58:01.352078 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 13:58:01.361032 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:01.361855 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:01.362624 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:01.368542 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:01.376914 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:01.378484 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:58:01.391304 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 13:58:01.396147 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:58:01.424093 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 13:58:01.427333 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 13:58:01.437340 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 13:58:01.522724 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 13:58:01.554210 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 13:58:01.584563 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 13:58:01.605870 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 13:58:01.606441 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:01.666904 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 13:58:01.669996 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 13:58:01.697420 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 13:58:01.766870 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 13:58:01.777230 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 13:58:01.783674 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:01.787233 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:01.788218 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:01.788568 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:01.794148 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:01.800324 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:01.835070 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:58:01.864465 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 13:58:01.865059 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 13:58:01.873884 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 13:58:01.936337 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 13:58:01.936805 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 13:58:01.980329 13118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240503 13:58:01.981142 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 13:58:01.989130 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 13:58:01.994424 15426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 118 similar messages]
W20240503 13:58:01.995124 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 13:58:02.029328 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:02.069286 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 13:58:02.084930 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 13:58:02.114274 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 13:58:02.123800 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 13:58:02.129670 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 13:58:02.153270 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 13:58:02.170225 13249 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 112 similar messages]
W20240503 13:58:02.174513 15691 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 108 similar messages]
W20240503 13:58:02.184860 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 13:58:02.201452 12987 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 73 similar messages]
I20240503 13:58:02.207643 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:02.208469 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:02.216001 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:02.226703 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:02.226814 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:02.229552 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:58:02.245451 15558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 88 similar messages]
I20240503 13:58:02.267529 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:58:02.274849 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 13:58:02.321225 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240503 13:58:02.322201 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 13:58:02.352964 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 13:58:02.376039 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 13:58:02.597478 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:02.603228 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 13:58:02.651937 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:02.656528 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:02.666687 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:02.666772 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:02.667320 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:58:02.685186 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 13:58:02.688803 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:02.689967 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:58:02.739813 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 13:58:02.778800 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 13:58:02.857460 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 13:58:02.915302 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 13:58:03.018864 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 13:58:03.060514 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 13:58:03.063699 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 13:58:03.064704 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:03.066556 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:03.078157 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:03.079346 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:03.084714 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:03.094316 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:03.113931 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:58:03.140326 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:03.165151 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 13:58:03.178267 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 13:58:03.191655 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 13:58:03.228670 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 13:58:03.259470 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 13:58:03.264518 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 13:58:03.271447 12896 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 6 similar messages]
W20240503 13:58:03.272181 12896 catalog_manager.cc:4573] TS e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2427783ee0e348b98453818704ece5f8: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:03.273835 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 13:58:03.337910 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 13:58:03.472410 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 13:58:03.492908 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:03.493156 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:03.494601 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:03.495038 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:03.499434 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:03.504894 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:58:03.519646 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 13:58:03.520779 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:58:03.521349 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 13:58:03.533788 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 13:58:03.577139 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 13:58:03.646468 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 13:58:03.668686 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 13:58:03.708442 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 13:58:03.714993 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 13:58:03.716562 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:03.763705 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 13:58:03.806622 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 13:58:03.857293 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 13:58:03.927812 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:03.937650 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:58:03.947412 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 13:58:03.952651 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:03.953040 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:03.955103 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:03.956367 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:58:03.984941 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 13:58:03.990566 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:58:04.022483 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 13:58:04.061358 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 13:58:04.086414 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 13:58:04.100119 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 13:58:04.106668 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 13:58:04.159679 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 13:58:04.169894 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 13:58:04.173134 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 13:58:04.189023 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 13:58:04.201084 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 13:58:04.220860 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:04.227435 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 13:58:04.263098 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 13:58:04.321538 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 13:58:04.346242 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 13:58:04.364575 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:04.365356 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:04.374946 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:04.375610 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:04.378150 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:04.380873 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:58:04.394876 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 13:58:04.400637 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:58:04.427737 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 13:58:04.432193 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 13:58:04.511910 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 13:58:04.568840 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 13:58:04.584429 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 13:58:04.613425 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 13:58:04.638999 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 13:58:04.704962 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 13:58:04.712191 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:04.732600 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 13:58:04.755190 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 13:58:04.774317 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:04.775938 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:04.779656 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:04.782905 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:04.795264 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:58:04.805225 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 13:58:04.814035 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 13:58:04.814502 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 13:58:04.833757 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:04.843160 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:58:04.848570 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 13:58:04.950208 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 13:58:05.059975 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 13:58:05.091931 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 13:58:05.097739 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 13:58:05.123669 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:05.216095 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 13:58:05.232810 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 13:58:05.239166 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:05.242878 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:05.247387 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:05.248445 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:05.252096 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:05.262728 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:05.290730 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:58:05.347952 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 13:58:05.402456 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 13:58:05.476830 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 13:58:05.519340 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:05.539749 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 13:58:05.540205 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 13:58:05.586567 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 13:58:05.632233 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 13:58:05.658529 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 13:58:05.669137 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:05.672766 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:05.676363 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:05.680665 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:05.684794 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:58:05.686492 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 13:58:05.687244 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 13:58:05.691970 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:05.717314 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:58:05.758487 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 13:58:05.758595 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 13:58:05.808267 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 13:58:05.844065 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 13:58:06.035720 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:06.066727 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 13:58:06.076596 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 13:58:06.086995 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:06.096413 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:06.096621 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:06.098017 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:06.109220 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:06.109382 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:06.109906 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:58:06.114102 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 13:58:06.148063 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 13:58:06.153092 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 13:58:06.246038 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 13:58:06.250367 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 13:58:06.310360 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 13:58:06.317471 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 13:58:06.350862 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 13:58:06.354764 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 13:58:06.354902 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 13:58:06.414132 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:06.426529 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 13:58:06.431624 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 13:58:06.460069 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 13:58:06.510794 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 13:58:06.531337 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:06.532446 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:58:06.532799 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 13:58:06.532845 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 13:58:06.536935 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:06.548707 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:06.552412 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:06.552687 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:58:06.554172 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 13:58:06.561856 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:58:06.590324 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 13:58:06.597342 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 13:58:06.717006 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 13:58:06.721704 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 13:58:06.729529 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 13:58:06.754994 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 13:58:06.813321 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 13:58:06.894884 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 13:58:06.946019 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 13:58:06.952865 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:06.989784 13118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 57 similar messages]
W20240503 13:58:06.991003 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 13:58:06.991364 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:06.996132 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:06.999861 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:06.999893 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:07.003484 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:07.004966 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:58:07.015309 15426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 120 similar messages]
I20240503 13:58:07.016841 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:58:07.076830 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 13:58:07.076910 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 13:58:07.087672 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 13:58:07.125821 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 13:58:07.132340 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 13:58:07.176113 13249 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 109 similar messages]
W20240503 13:58:07.176836 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 13:58:07.238193 12987 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 67 similar messages]
W20240503 13:58:07.238849 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 13:58:07.280607 15691 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 109 similar messages]
W20240503 13:58:07.305186 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 13:58:07.308138 15558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 93 similar messages]
W20240503 13:58:07.335482 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 13:58:07.341660 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 13:58:07.394665 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 13:58:07.414851 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:07.420742 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:07.421052 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:07.421494 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:07.425904 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:58:07.444795 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 13:58:07.446647 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240503 13:58:07.447530 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:58:07.448257 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:07.448248 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:58:07.478418 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 13:58:07.504707 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 13:58:07.518896 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 13:58:07.590394 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 13:58:07.665517 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 13:58:07.719821 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 13:58:07.762410 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 13:58:07.764325 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 13:58:07.796222 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 13:58:07.829016 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 13:58:07.836391 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:07.843097 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:07.848117 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:07.848219 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:07.850878 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:58:07.859345 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:58:07.877924 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:07.881011 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 13:58:07.890385 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 13:58:07.919183 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 13:58:07.944702 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 13:58:07.968619 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 13:58:08.220042 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 13:58:08.252996 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 13:58:08.274711 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 13:58:08.296986 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:08.304286 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:08.305805 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:08.311980 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:58:08.329877 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 13:58:08.332902 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:08.345976 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:58:08.348673 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 13:58:08.361168 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:58:08.433856 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 13:58:08.437572 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:08.473040 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 13:58:08.518556 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 13:58:08.546284 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 13:58:08.556820 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 13:58:08.575232 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 13:58:08.579846 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 13:58:08.686718 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 13:58:08.699671 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 13:58:08.725759 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:08.728966 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:08.729116 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:08.740443 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:08.742779 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:58:08.748720 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 13:58:08.753607 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:08.759547 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:58:08.780158 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 13:58:08.807212 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 13:58:08.819480 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 13:58:08.839545 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 13:58:08.846702 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 13:58:08.850844 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 13:58:08.914059 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 13:58:08.945077 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:08.945093 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 13:58:08.949075 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 13:58:08.987874 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 13:58:09.068655 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 13:58:09.115708 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 13:58:09.138125 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 13:58:09.146703 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 13:58:09.168416 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:09.171514 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:09.172650 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:09.178002 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:09.186417 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:09.186697 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:58:09.186918 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 13:58:09.192124 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:58:09.324090 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 13:58:09.350268 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 13:58:09.440057 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 13:58:09.472182 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 13:58:09.473239 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 13:58:09.510322 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 13:58:09.513361 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 13:58:09.548877 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 13:58:09.561642 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 13:58:09.567386 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:58:09.617779 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:09.619364 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:09.621723 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:09.622334 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:58:09.627642 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 13:58:09.633450 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:09.641232 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:09.641403 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:58:09.682758 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 13:58:09.748517 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 13:58:09.751837 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 13:58:09.758846 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 13:58:09.777699 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 13:58:09.780335 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 13:58:09.825907 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 13:58:09.826295 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 13:58:09.964546 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 13:58:09.988734 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 13:58:10.004714 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 13:58:10.035720 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:10.042037 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:10.042510 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:10.043550 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:10.070639 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:10.076817 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:10.096078 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:58:10.114986 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:10.184789 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 13:58:10.195438 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 13:58:10.215955 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 13:58:10.268872 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 13:58:10.296259 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 13:58:10.296629 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 13:58:10.300665 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 13:58:10.313506 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 13:58:10.362193 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 13:58:10.362990 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 13:58:10.370747 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 13:58:10.476760 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:10.481583 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:10.483175 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:10.484787 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:10.491616 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:10.498838 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:10.503762 12901 maintenance_manager.cc:419] P f21c260fa03f4452870a21b3091e1130: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033384
I20240503 13:58:10.504707 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:58:10.514371 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:10.514653 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 13:58:10.526234 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 13:58:10.538537 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 13:58:10.570119 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 13:58:10.573949 12900 maintenance_manager.cc:643] P f21c260fa03f4452870a21b3091e1130: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.070s user 0.033s sys 0.007s Metrics: {"bytes_written":111488,"cfile_init":1,"dirs.queue_time_us":11867,"dirs.run_cpu_time_us":226,"dirs.run_wall_time_us":1223,"drs_written":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1241,"lbm_writes_lt_1ms":34,"peak_mem_usage":0,"rows_written":313,"thread_start_us":177,"threads_started":2,"wal-append.queue_time_us":1337}
I20240503 13:58:10.575062 12901 maintenance_manager.cc:419] P f21c260fa03f4452870a21b3091e1130: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 34942 bytes on disk
I20240503 13:58:10.575727 12900 maintenance_manager.cc:643] P f21c260fa03f4452870a21b3091e1130: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4}
I20240503 13:58:10.920053 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:58:10.920523 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 13:58:10.922626 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 13:58:10.923600 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:10.925573 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:10.927095 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:10.934691 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:10.935276 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:10.948665 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:58:10.991552 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:11.007061 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 13:58:11.029008 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 13:58:11.092813 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 13:58:11.136921 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 13:58:11.137379 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 13:58:11.240309 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 13:58:11.250620 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 13:58:11.277798 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 13:58:11.297235 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 13:58:11.308032 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 13:58:11.314500 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 13:58:11.316136 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 13:58:11.324270 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 13:58:11.326424 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 13:58:11.366935 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:11.368629 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:11.369354 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:11.375424 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:11.375748 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:58:11.375917 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 13:58:11.387405 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:58:11.391206 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 13:58:11.400010 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 13:58:11.412685 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:58:11.419584 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:11.442615 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 13:58:11.481678 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 13:58:11.589052 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 13:58:11.591066 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 13:58:11.600953 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 13:58:11.700634 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 13:58:11.796420 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:58:11.796633 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 13:58:11.799270 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:11.800382 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:11.811262 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:11.813279 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:11.813601 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:11.830755 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:58:11.842867 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 13:58:11.880564 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:11.882002 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 13:58:11.937426 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 13:58:11.946422 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 13:58:11.961525 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 13:58:11.975319 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 13:58:12.001616 13118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240503 13:58:12.016640 15426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240503 13:58:12.081673 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 13:58:12.090930 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 13:58:12.102569 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 13:58:12.167104 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 13:58:12.170176 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 13:58:12.216043 13249 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 110 similar messages]
W20240503 13:58:12.218830 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 13:58:12.235682 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 13:58:12.237947 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:12.244863 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:12.255081 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:12.259549 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:12.260668 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:12.261572 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:12.272358 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 13:58:12.290138 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 13:58:12.291694 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 13:58:12.307318 15691 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 112 similar messages]
W20240503 13:58:12.311072 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 13:58:12.312609 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 13:58:12.324038 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:12.370011 12987 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240503 13:58:12.438668 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 13:58:12.450444 15558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240503 13:58:12.451126 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 13:58:12.502650 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 13:58:12.506827 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 13:58:12.537035 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240503 13:58:12.565589 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 13:58:12.596189 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 13:58:12.663645 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:12.667027 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:12.669566 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:58:12.675118 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 13:58:12.676661 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 13:58:12.680915 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:12.684698 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:12.686743 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:12.687983 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:58:12.723522 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 13:58:12.734570 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 13:58:12.755146 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 13:58:12.765033 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 13:58:12.800336 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 13:58:12.858954 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 13:58:12.861058 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 13:58:12.865298 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:12.933033 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 13:58:13.043265 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 13:58:13.074146 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:13.078014 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:13.080392 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:13.080619 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:13.082000 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:13.091962 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:13.119603 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:58:13.245317 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 13:58:13.314568 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 13:58:13.331643 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 13:58:13.371723 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 13:58:13.399528 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:13.429939 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 13:58:13.442814 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 13:58:13.489008 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:13.489632 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:13.492393 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:13.495290 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:13.504645 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:13.505605 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:13.506085 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:58:13.533447 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 13:58:13.578145 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 13:58:13.587905 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 13:58:13.595985 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 13:58:13.633283 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 13:58:13.634846 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 13:58:13.666898 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 13:58:13.728312 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 13:58:13.820729 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 13:58:13.834968 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 13:58:13.850243 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 13:58:13.887359 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:13.931735 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 13:58:13.933190 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:13.933760 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:13.939746 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:58:13.946353 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 13:58:13.946499 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 13:58:13.947227 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:13.956923 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:13.962558 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:13.966387 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:58:14.002461 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 13:58:14.035784 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 13:58:14.082777 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 13:58:14.090695 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 13:58:14.152215 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 13:58:14.159521 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 13:58:14.236584 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 13:58:14.284229 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 13:58:14.293633 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 13:58:14.321767 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 13:58:14.364518 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:14.374714 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:14.377018 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:14.377645 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:58:14.381210 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 13:58:14.385579 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:14.389106 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:14.393532 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:58:14.420713 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:14.421248 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 13:58:14.432690 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 13:58:14.476974 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 13:58:14.493228 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 13:58:14.520469 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 13:58:14.523783 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 13:58:14.534349 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 13:58:14.640236 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 13:58:14.665930 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 13:58:14.721555 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 13:58:14.732182 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 13:58:14.736354 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 13:58:14.759497 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 13:58:14.778982 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:14.793337 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:14.793651 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:14.795934 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:14.796674 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:14.808689 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:14.821094 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 13:58:14.830363 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:14.850421 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 13:58:14.875977 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 13:58:14.904095 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 13:58:14.924885 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 13:58:15.077535 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 13:58:15.088752 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 13:58:15.178390 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 13:58:15.203483 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:15.213956 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:15.216476 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:15.217222 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:15.222452 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:15.234278 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:58:15.240585 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 13:58:15.249307 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 13:58:15.258863 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 13:58:15.262890 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:58:15.271239 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:15.276595 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 13:58:15.309952 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 13:58:15.316130 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 13:58:15.456784 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 13:58:15.489815 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 13:58:15.489832 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 13:58:15.489913 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 13:58:15.578078 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 13:58:15.591809 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 13:58:15.634127 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 13:58:15.645967 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:15.651777 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:15.656615 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:15.657250 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:15.663228 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:15.682898 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:15.693100 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:58:15.694635 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:15.710095 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 13:58:15.862044 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 13:58:15.892177 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 13:58:15.984026 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 13:58:15.986325 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 13:58:15.989938 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 13:58:16.081501 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:16.082562 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:16.088357 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:58:16.092798 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 13:58:16.097123 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:16.109998 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:16.110680 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:58:16.113214 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 13:58:16.121014 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:58:16.156309 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 13:58:16.194123 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:16.221791 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 13:58:16.229485 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 13:58:16.235262 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 13:58:16.282778 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 13:58:16.298637 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 13:58:16.361862 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 13:58:16.417680 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 13:58:16.462348 13363 maintenance_manager.cc:419] P ba0194098aa84449ae17e8a0d4c43703: Scheduling FlushMRSOp(477c33f203f649019eff2b28524fad90): perf score=0.033352
W20240503 13:58:16.475808 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 13:58:16.487936 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 13:58:16.512331 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 13:58:16.513384 13101 maintenance_manager.cc:419] P 60ba3ffe3b7746469834cd91c84be14e: Scheduling FlushMRSOp(df28831df92d4c12b63b2f7da6c6449a): perf score=0.033369
I20240503 13:58:16.530018 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:16.534927 13232 maintenance_manager.cc:419] P 8ccf5a4ac450414c9d3579bd5800b770: Scheduling FlushMRSOp(7239f94a7c88433aa6062390a54e7562): perf score=0.033378
W20240503 13:58:16.539660 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 13:58:16.544334 13253 maintenance_manager.cc:643] P ba0194098aa84449ae17e8a0d4c43703: FlushMRSOp(477c33f203f649019eff2b28524fad90) complete. Timing: real 0.081s user 0.026s sys 0.005s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":173,"drs_written":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4,"lbm_write_time_us":641,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":215,"threads_started":3,"wal-append.queue_time_us":12124}
I20240503 13:58:16.546957 13363 maintenance_manager.cc:419] P ba0194098aa84449ae17e8a0d4c43703: Scheduling UndoDeltaBlockGCOp(477c33f203f649019eff2b28524fad90): 4583 bytes on disk
I20240503 13:58:16.548378 13253 maintenance_manager.cc:643] P ba0194098aa84449ae17e8a0d4c43703: UndoDeltaBlockGCOp(477c33f203f649019eff2b28524fad90) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4}
W20240503 13:58:16.572405 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 13:58:16.575026 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:16.581363 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:16.583861 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:16.590025 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:16.590938 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:16.595813 13494 maintenance_manager.cc:419] P 7690ee79577c45e3a2c8f8b552c0bff8: Scheduling FlushMRSOp(7239f94a7c88433aa6062390a54e7562): perf score=0.033395
W20240503 13:58:16.602281 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 13:58:16.602393 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 13:58:16.643776 12991 maintenance_manager.cc:643] P 60ba3ffe3b7746469834cd91c84be14e: FlushMRSOp(df28831df92d4c12b63b2f7da6c6449a) complete. Timing: real 0.130s user 0.036s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":15234,"dirs.queue_time_us":8500,"dirs.run_cpu_time_us":224,"dirs.run_wall_time_us":3259,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":582,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":263,"threads_started":3,"wal-append.queue_time_us":4394}
I20240503 13:58:16.649705 13101 maintenance_manager.cc:419] P 60ba3ffe3b7746469834cd91c84be14e: Scheduling UndoDeltaBlockGCOp(df28831df92d4c12b63b2f7da6c6449a): 4838 bytes on disk
W20240503 13:58:16.655920 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 13:58:16.656976 12991 maintenance_manager.cc:643] P 60ba3ffe3b7746469834cd91c84be14e: UndoDeltaBlockGCOp(df28831df92d4c12b63b2f7da6c6449a) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":854,"lbm_reads_lt_1ms":4}
W20240503 13:58:16.673916 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 13:58:16.720675 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:58:16.723172 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 13:58:16.727358 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:58:16.803457 13384 maintenance_manager.cc:643] P 7690ee79577c45e3a2c8f8b552c0bff8: FlushMRSOp(7239f94a7c88433aa6062390a54e7562) complete. Timing: real 0.207s user 0.031s sys 0.005s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":36986,"dirs.queue_time_us":15710,"dirs.run_cpu_time_us":205,"dirs.run_wall_time_us":1130,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1514,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":438,"threads_started":3,"wal-append.queue_time_us":74052}
I20240503 13:58:16.805737 13494 maintenance_manager.cc:419] P 7690ee79577c45e3a2c8f8b552c0bff8: Scheduling FlushMRSOp(477c33f203f649019eff2b28524fad90): perf score=0.033452
W20240503 13:58:16.839461 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 13:58:16.839568 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 13:58:16.852247 13384 maintenance_manager.cc:643] P 7690ee79577c45e3a2c8f8b552c0bff8: FlushMRSOp(477c33f203f649019eff2b28524fad90) complete. Timing: real 0.046s user 0.018s sys 0.001s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":6238,"dirs.run_cpu_time_us":197,"dirs.run_wall_time_us":1523,"drs_written":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4,"lbm_write_time_us":576,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":111,"threads_started":1,"wal-append.queue_time_us":12186}
I20240503 13:58:16.854251 13494 maintenance_manager.cc:419] P 7690ee79577c45e3a2c8f8b552c0bff8: Scheduling UndoDeltaBlockGCOp(7239f94a7c88433aa6062390a54e7562): 4923 bytes on disk
W20240503 13:58:16.856789 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 13:58:16.856804 13384 maintenance_manager.cc:643] P 7690ee79577c45e3a2c8f8b552c0bff8: UndoDeltaBlockGCOp(7239f94a7c88433aa6062390a54e7562) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4}
I20240503 13:58:16.862390 13494 maintenance_manager.cc:419] P 7690ee79577c45e3a2c8f8b552c0bff8: Scheduling UndoDeltaBlockGCOp(477c33f203f649019eff2b28524fad90): 4583 bytes on disk
I20240503 13:58:16.862859 13384 maintenance_manager.cc:643] P 7690ee79577c45e3a2c8f8b552c0bff8: UndoDeltaBlockGCOp(477c33f203f649019eff2b28524fad90) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4}
I20240503 13:58:16.874332 13122 maintenance_manager.cc:643] P 8ccf5a4ac450414c9d3579bd5800b770: FlushMRSOp(7239f94a7c88433aa6062390a54e7562) complete. Timing: real 0.339s user 0.031s sys 0.005s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":3172,"dirs.queue_time_us":17630,"dirs.run_cpu_time_us":199,"dirs.run_wall_time_us":2564,"drs_written":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4,"lbm_write_time_us":723,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":999,"threads_started":3,"wal-append.queue_time_us":12458}
I20240503 13:58:16.877311 13232 maintenance_manager.cc:419] P 8ccf5a4ac450414c9d3579bd5800b770: Scheduling FlushMRSOp(df28831df92d4c12b63b2f7da6c6449a): perf score=0.033467
W20240503 13:58:16.884151 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 13:58:16.908800 13122 maintenance_manager.cc:643] P 8ccf5a4ac450414c9d3579bd5800b770: FlushMRSOp(df28831df92d4c12b63b2f7da6c6449a) complete. Timing: real 0.031s user 0.019s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":165,"dirs.run_cpu_time_us":191,"dirs.run_wall_time_us":1234,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":568,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":138,"threads_started":1,"wal-append.queue_time_us":240}
I20240503 13:58:16.911126 13232 maintenance_manager.cc:419] P 8ccf5a4ac450414c9d3579bd5800b770: Scheduling UndoDeltaBlockGCOp(7239f94a7c88433aa6062390a54e7562): 4923 bytes on disk
I20240503 13:58:16.911702 13122 maintenance_manager.cc:643] P 8ccf5a4ac450414c9d3579bd5800b770: UndoDeltaBlockGCOp(7239f94a7c88433aa6062390a54e7562) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4}
I20240503 13:58:16.915087 13232 maintenance_manager.cc:419] P 8ccf5a4ac450414c9d3579bd5800b770: Scheduling UndoDeltaBlockGCOp(df28831df92d4c12b63b2f7da6c6449a): 4838 bytes on disk
I20240503 13:58:16.915561 13122 maintenance_manager.cc:643] P 8ccf5a4ac450414c9d3579bd5800b770: UndoDeltaBlockGCOp(df28831df92d4c12b63b2f7da6c6449a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
W20240503 13:58:16.916551 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 13:58:16.976433 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 13:58:16.983875 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 13:58:16.988580 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 13:58:17.050850 15426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240503 13:58:17.111636 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 13:58:17.116269 13118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240503 13:58:17.116643 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 13:58:17.173118 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 13:58:17.178583 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:17.183836 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:58:17.188227 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 13:58:17.191908 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:17.192031 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:17.194689 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:58:17.203222 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 13:58:17.209141 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:58:17.210973 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:17.252434 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:58:17.259255 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 13:58:17.265100 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 13:58:17.305509 13363 maintenance_manager.cc:419] P ba0194098aa84449ae17e8a0d4c43703: Scheduling FlushMRSOp(a49b6f596d724b35b757ee69fe15e5a5): perf score=0.033357
W20240503 13:58:17.314426 15691 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 109 similar messages]
I20240503 13:58:17.320356 13253 maintenance_manager.cc:643] P ba0194098aa84449ae17e8a0d4c43703: FlushMRSOp(a49b6f596d724b35b757ee69fe15e5a5) complete. Timing: real 0.015s user 0.013s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":157,"dirs.run_cpu_time_us":188,"dirs.run_wall_time_us":1645,"drs_written":1,"lbm_read_time_us":43,"lbm_reads_lt_1ms":4,"lbm_write_time_us":489,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":140,"threads_started":2,"wal-append.queue_time_us":159}
I20240503 13:58:17.322191 13363 maintenance_manager.cc:419] P ba0194098aa84449ae17e8a0d4c43703: Scheduling UndoDeltaBlockGCOp(a49b6f596d724b35b757ee69fe15e5a5): 4923 bytes on disk
I20240503 13:58:17.322614 13253 maintenance_manager.cc:643] P ba0194098aa84449ae17e8a0d4c43703: UndoDeltaBlockGCOp(a49b6f596d724b35b757ee69fe15e5a5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4}
I20240503 13:58:17.324221 13363 maintenance_manager.cc:419] P ba0194098aa84449ae17e8a0d4c43703: Scheduling FlushMRSOp(c047c61f42f04484842125847ca677aa): perf score=0.033361
I20240503 13:58:17.337514 13253 maintenance_manager.cc:643] P ba0194098aa84449ae17e8a0d4c43703: FlushMRSOp(c047c61f42f04484842125847ca677aa) complete. Timing: real 0.013s user 0.012s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":39,"dirs.run_cpu_time_us":140,"dirs.run_wall_time_us":1319,"drs_written":1,"lbm_read_time_us":38,"lbm_reads_lt_1ms":4,"lbm_write_time_us":499,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":135,"threads_started":1,"wal-append.queue_time_us":218}
I20240503 13:58:17.339222 13363 maintenance_manager.cc:419] P ba0194098aa84449ae17e8a0d4c43703: Scheduling FlushMRSOp(4647faf6a457415d8ae2b6691f452eab): perf score=0.033362
I20240503 13:58:17.352718 13253 maintenance_manager.cc:643] P ba0194098aa84449ae17e8a0d4c43703: FlushMRSOp(4647faf6a457415d8ae2b6691f452eab) complete. Timing: real 0.013s user 0.013s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":41,"dirs.run_cpu_time_us":177,"dirs.run_wall_time_us":1364,"drs_written":1,"lbm_read_time_us":34,"lbm_reads_lt_1ms":4,"lbm_write_time_us":508,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":118,"threads_started":1,"wal-append.queue_time_us":192}
W20240503 13:58:17.354379 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 13:58:17.354701 13363 maintenance_manager.cc:419] P ba0194098aa84449ae17e8a0d4c43703: Scheduling UndoDeltaBlockGCOp(4647faf6a457415d8ae2b6691f452eab): 4838 bytes on disk
I20240503 13:58:17.355101 13253 maintenance_manager.cc:643] P ba0194098aa84449ae17e8a0d4c43703: UndoDeltaBlockGCOp(4647faf6a457415d8ae2b6691f452eab) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4}
W20240503 13:58:17.356418 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 13:58:17.356580 13363 maintenance_manager.cc:419] P ba0194098aa84449ae17e8a0d4c43703: Scheduling UndoDeltaBlockGCOp(c047c61f42f04484842125847ca677aa): 4583 bytes on disk
I20240503 13:58:17.356952 13253 maintenance_manager.cc:643] P ba0194098aa84449ae17e8a0d4c43703: UndoDeltaBlockGCOp(c047c61f42f04484842125847ca677aa) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
W20240503 13:58:17.364146 13249 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 110 similar messages]
I20240503 13:58:17.368150 13494 maintenance_manager.cc:419] P 7690ee79577c45e3a2c8f8b552c0bff8: Scheduling FlushMRSOp(c047c61f42f04484842125847ca677aa): perf score=0.033370
I20240503 13:58:17.382958 13384 maintenance_manager.cc:643] P 7690ee79577c45e3a2c8f8b552c0bff8: FlushMRSOp(c047c61f42f04484842125847ca677aa) complete. Timing: real 0.015s user 0.011s sys 0.002s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":185,"dirs.run_cpu_time_us":182,"dirs.run_wall_time_us":1712,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":597,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":154,"threads_started":2,"wal-append.queue_time_us":153}
I20240503 13:58:17.384461 13494 maintenance_manager.cc:419] P 7690ee79577c45e3a2c8f8b552c0bff8: Scheduling FlushMRSOp(4647faf6a457415d8ae2b6691f452eab): perf score=0.033374
I20240503 13:58:17.398916 13384 maintenance_manager.cc:643] P 7690ee79577c45e3a2c8f8b552c0bff8: FlushMRSOp(4647faf6a457415d8ae2b6691f452eab) complete. Timing: real 0.014s user 0.012s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":94,"dirs.run_cpu_time_us":217,"dirs.run_wall_time_us":1766,"drs_written":1,"lbm_read_time_us":35,"lbm_reads_lt_1ms":4,"lbm_write_time_us":439,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":76,"threads_started":1,"wal-append.queue_time_us":242}
I20240503 13:58:17.400377 13494 maintenance_manager.cc:419] P 7690ee79577c45e3a2c8f8b552c0bff8: Scheduling UndoDeltaBlockGCOp(4647faf6a457415d8ae2b6691f452eab): 4838 bytes on disk
I20240503 13:58:17.400843 13384 maintenance_manager.cc:643] P 7690ee79577c45e3a2c8f8b552c0bff8: UndoDeltaBlockGCOp(4647faf6a457415d8ae2b6691f452eab) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
I20240503 13:58:17.402103 13494 maintenance_manager.cc:419] P 7690ee79577c45e3a2c8f8b552c0bff8: Scheduling UndoDeltaBlockGCOp(c047c61f42f04484842125847ca677aa): 4583 bytes on disk
I20240503 13:58:17.402462 13384 maintenance_manager.cc:643] P 7690ee79577c45e3a2c8f8b552c0bff8: UndoDeltaBlockGCOp(c047c61f42f04484842125847ca677aa) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4}
I20240503 13:58:17.413268 13101 maintenance_manager.cc:419] P 60ba3ffe3b7746469834cd91c84be14e: Scheduling FlushMRSOp(c047c61f42f04484842125847ca677aa): perf score=0.033383
I20240503 13:58:17.424774 13232 maintenance_manager.cc:419] P 8ccf5a4ac450414c9d3579bd5800b770: Scheduling FlushMRSOp(a49b6f596d724b35b757ee69fe15e5a5): perf score=0.033387
W20240503 13:58:17.426898 12987 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 71 similar messages]
I20240503 13:58:17.427668 12991 maintenance_manager.cc:643] P 60ba3ffe3b7746469834cd91c84be14e: FlushMRSOp(c047c61f42f04484842125847ca677aa) complete. Timing: real 0.014s user 0.013s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":166,"dirs.run_cpu_time_us":222,"dirs.run_wall_time_us":1762,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":585,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":159,"threads_started":2,"wal-append.queue_time_us":161}
I20240503 13:58:17.429136 13101 maintenance_manager.cc:419] P 60ba3ffe3b7746469834cd91c84be14e: Scheduling UndoDeltaBlockGCOp(c047c61f42f04484842125847ca677aa): 4583 bytes on disk
I20240503 13:58:17.429500 12991 maintenance_manager.cc:643] P 60ba3ffe3b7746469834cd91c84be14e: UndoDeltaBlockGCOp(c047c61f42f04484842125847ca677aa) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4}
W20240503 13:58:17.434337 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 13:58:17.439393 13122 maintenance_manager.cc:643] P 8ccf5a4ac450414c9d3579bd5800b770: FlushMRSOp(a49b6f596d724b35b757ee69fe15e5a5) complete. Timing: real 0.014s user 0.012s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":197,"dirs.run_cpu_time_us":268,"dirs.run_wall_time_us":1682,"drs_written":1,"lbm_read_time_us":38,"lbm_reads_lt_1ms":4,"lbm_write_time_us":543,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":167,"threads_started":2,"wal-append.queue_time_us":160}
I20240503 13:58:17.441246 13232 maintenance_manager.cc:419] P 8ccf5a4ac450414c9d3579bd5800b770: Scheduling UndoDeltaBlockGCOp(a49b6f596d724b35b757ee69fe15e5a5): 4923 bytes on disk
I20240503 13:58:17.441690 13122 maintenance_manager.cc:643] P 8ccf5a4ac450414c9d3579bd5800b770: UndoDeltaBlockGCOp(a49b6f596d724b35b757ee69fe15e5a5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4}
W20240503 13:58:17.452349 15558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240503 13:58:17.486384 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 13:58:17.497149 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 13:58:17.517360 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 13:58:17.540576 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 13:58:17.546761 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 13:58:17.574401 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240503 13:58:17.575208 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:58:17.634375 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:17.641083 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:17.645073 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:17.645547 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:17.658519 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:17.660318 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:17.670542 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:58:17.719030 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 13:58:17.758833 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 13:58:17.796022 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 13:58:17.804688 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 13:58:17.818958 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 13:58:17.893101 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 13:58:18.017720 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 13:58:18.032459 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 13:58:18.050505 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:18.057828 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:18.058148 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:58:18.060902 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 13:58:18.062620 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:58:18.069335 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:18.070219 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:18.078222 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:58:18.090342 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 13:58:18.092017 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:58:18.121169 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 13:58:18.140339 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 13:58:18.199745 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 13:58:18.295780 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 13:58:18.359372 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 13:58:18.384631 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 13:58:18.384912 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 13:58:18.440891 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 13:58:18.469822 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:18.480810 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:18.483031 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:18.483418 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:18.484707 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:18.488078 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:18.501221 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:58:18.553339 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 13:58:18.565740 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 13:58:18.592793 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:18.607053 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 13:58:18.617586 13363 maintenance_manager.cc:419] P ba0194098aa84449ae17e8a0d4c43703: Scheduling FlushMRSOp(bb4f0bb47a29437090750c4486df2f02): perf score=0.033377
I20240503 13:58:18.630465 13253 maintenance_manager.cc:643] P ba0194098aa84449ae17e8a0d4c43703: FlushMRSOp(bb4f0bb47a29437090750c4486df2f02) complete. Timing: real 0.013s user 0.011s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":198,"dirs.run_cpu_time_us":220,"dirs.run_wall_time_us":1706,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":468,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"spinlock_wait_cycles":896,"thread_start_us":151,"threads_started":2,"wal-append.queue_time_us":140}
I20240503 13:58:18.631966 13363 maintenance_manager.cc:419] P ba0194098aa84449ae17e8a0d4c43703: Scheduling FlushMRSOp(b47b28a4766a425e8b9bdfea882f979d): perf score=0.033379
I20240503 13:58:18.645059 13253 maintenance_manager.cc:643] P ba0194098aa84449ae17e8a0d4c43703: FlushMRSOp(b47b28a4766a425e8b9bdfea882f979d) complete. Timing: real 0.013s user 0.010s sys 0.003s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":38,"dirs.run_cpu_time_us":186,"dirs.run_wall_time_us":3549,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":499,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":83,"threads_started":1,"wal-append.queue_time_us":277}
I20240503 13:58:18.646557 13363 maintenance_manager.cc:419] P ba0194098aa84449ae17e8a0d4c43703: Scheduling UndoDeltaBlockGCOp(bb4f0bb47a29437090750c4486df2f02): 4923 bytes on disk
I20240503 13:58:18.646903 13253 maintenance_manager.cc:643] P ba0194098aa84449ae17e8a0d4c43703: UndoDeltaBlockGCOp(bb4f0bb47a29437090750c4486df2f02) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4}
I20240503 13:58:18.648430 13363 maintenance_manager.cc:419] P ba0194098aa84449ae17e8a0d4c43703: Scheduling UndoDeltaBlockGCOp(b47b28a4766a425e8b9bdfea882f979d): 4836 bytes on disk
I20240503 13:58:18.648996 13253 maintenance_manager.cc:643] P ba0194098aa84449ae17e8a0d4c43703: UndoDeltaBlockGCOp(b47b28a4766a425e8b9bdfea882f979d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4}
W20240503 13:58:18.649924 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 13:58:18.661288 13494 maintenance_manager.cc:419] P 7690ee79577c45e3a2c8f8b552c0bff8: Scheduling FlushMRSOp(9eb0d911701b4b2bb33a6b7ab59f08aa): perf score=0.033388
W20240503 13:58:18.670626 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 13:58:18.674423 13384 maintenance_manager.cc:643] P 7690ee79577c45e3a2c8f8b552c0bff8: FlushMRSOp(9eb0d911701b4b2bb33a6b7ab59f08aa) complete. Timing: real 0.013s user 0.011s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":250,"dirs.run_cpu_time_us":192,"dirs.run_wall_time_us":1485,"drs_written":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4,"lbm_write_time_us":502,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":166,"threads_started":2,"wal-append.queue_time_us":137}
I20240503 13:58:18.675908 13494 maintenance_manager.cc:419] P 7690ee79577c45e3a2c8f8b552c0bff8: Scheduling UndoDeltaBlockGCOp(9eb0d911701b4b2bb33a6b7ab59f08aa): 4583 bytes on disk
I20240503 13:58:18.676357 13384 maintenance_manager.cc:643] P 7690ee79577c45e3a2c8f8b552c0bff8: UndoDeltaBlockGCOp(9eb0d911701b4b2bb33a6b7ab59f08aa) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
I20240503 13:58:18.688732 13101 maintenance_manager.cc:419] P 60ba3ffe3b7746469834cd91c84be14e: Scheduling FlushMRSOp(b47b28a4766a425e8b9bdfea882f979d): perf score=0.033396
W20240503 13:58:18.691614 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 13:58:18.702220 13232 maintenance_manager.cc:419] P 8ccf5a4ac450414c9d3579bd5800b770: Scheduling FlushMRSOp(bb4f0bb47a29437090750c4486df2f02): perf score=0.033403
I20240503 13:58:18.703320 12991 maintenance_manager.cc:643] P 60ba3ffe3b7746469834cd91c84be14e: FlushMRSOp(b47b28a4766a425e8b9bdfea882f979d) complete. Timing: real 0.014s user 0.012s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":179,"dirs.run_cpu_time_us":183,"dirs.run_wall_time_us":1563,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":469,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":149,"threads_started":2,"wal-append.queue_time_us":150}
I20240503 13:58:18.705250 13101 maintenance_manager.cc:419] P 60ba3ffe3b7746469834cd91c84be14e: Scheduling UndoDeltaBlockGCOp(b47b28a4766a425e8b9bdfea882f979d): 4836 bytes on disk
I20240503 13:58:18.705766 12991 maintenance_manager.cc:643] P 60ba3ffe3b7746469834cd91c84be14e: UndoDeltaBlockGCOp(b47b28a4766a425e8b9bdfea882f979d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":116,"lbm_reads_lt_1ms":4}
I20240503 13:58:18.715838 13122 maintenance_manager.cc:643] P 8ccf5a4ac450414c9d3579bd5800b770: FlushMRSOp(bb4f0bb47a29437090750c4486df2f02) complete. Timing: real 0.013s user 0.012s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":158,"dirs.run_cpu_time_us":171,"dirs.run_wall_time_us":1375,"drs_written":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4,"lbm_write_time_us":439,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":126,"threads_started":2,"wal-append.queue_time_us":119}
I20240503 13:58:18.717496 13232 maintenance_manager.cc:419] P 8ccf5a4ac450414c9d3579bd5800b770: Scheduling FlushMRSOp(9eb0d911701b4b2bb33a6b7ab59f08aa): perf score=0.033406
I20240503 13:58:18.729470 13122 maintenance_manager.cc:643] P 8ccf5a4ac450414c9d3579bd5800b770: FlushMRSOp(9eb0d911701b4b2bb33a6b7ab59f08aa) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":46,"dirs.run_cpu_time_us":175,"dirs.run_wall_time_us":1544,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":389,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":92,"threads_started":1,"wal-append.queue_time_us":178}
I20240503 13:58:18.731017 13232 maintenance_manager.cc:419] P 8ccf5a4ac450414c9d3579bd5800b770: Scheduling UndoDeltaBlockGCOp(bb4f0bb47a29437090750c4486df2f02): 4923 bytes on disk
I20240503 13:58:18.731405 13122 maintenance_manager.cc:643] P 8ccf5a4ac450414c9d3579bd5800b770: UndoDeltaBlockGCOp(bb4f0bb47a29437090750c4486df2f02) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
I20240503 13:58:18.732721 13232 maintenance_manager.cc:419] P 8ccf5a4ac450414c9d3579bd5800b770: Scheduling UndoDeltaBlockGCOp(9eb0d911701b4b2bb33a6b7ab59f08aa): 4583 bytes on disk
I20240503 13:58:18.733072 13122 maintenance_manager.cc:643] P 8ccf5a4ac450414c9d3579bd5800b770: UndoDeltaBlockGCOp(9eb0d911701b4b2bb33a6b7ab59f08aa) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4}
W20240503 13:58:18.763510 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 13:58:18.770576 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 13:58:18.896831 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:18.898053 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:18.908663 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:18.908957 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:58:18.910431 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 13:58:18.912703 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:58:18.923378 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 13:58:18.923970 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 13:58:18.962057 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:18.968652 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:58:19.015996 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 13:58:19.027179 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 13:58:19.028049 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:19.069128 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 13:58:19.084323 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 13:58:19.125974 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 13:58:19.145788 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 13:58:19.147042 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 13:58:19.235612 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 13:58:19.244040 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 13:58:19.264703 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 13:58:19.269757 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 13:58:19.299321 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 13:58:19.344238 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 13:58:19.345361 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 13:58:19.353176 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:19.356920 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:19.356997 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:58:19.360455 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 13:58:19.373710 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:19.385252 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:19.396762 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:58:19.418220 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 13:58:19.420755 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:19.434522 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 13:58:19.436961 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 13:58:19.519625 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 13:58:19.539988 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 13:58:19.657727 13363 maintenance_manager.cc:419] P ba0194098aa84449ae17e8a0d4c43703: Scheduling FlushMRSOp(258d1ec1f0694c15a4d41bd064f35b4e): perf score=0.033335
I20240503 13:58:19.675097 13253 maintenance_manager.cc:643] P ba0194098aa84449ae17e8a0d4c43703: FlushMRSOp(258d1ec1f0694c15a4d41bd064f35b4e) complete. Timing: real 0.017s user 0.012s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":173,"dirs.run_cpu_time_us":176,"dirs.run_wall_time_us":1596,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":479,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":147,"threads_started":2,"wal-append.queue_time_us":153}
I20240503 13:58:19.676978 13363 maintenance_manager.cc:419] P ba0194098aa84449ae17e8a0d4c43703: Scheduling UndoDeltaBlockGCOp(258d1ec1f0694c15a4d41bd064f35b4e): 4838 bytes on disk
I20240503 13:58:19.677515 13253 maintenance_manager.cc:643] P ba0194098aa84449ae17e8a0d4c43703: UndoDeltaBlockGCOp(258d1ec1f0694c15a4d41bd064f35b4e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
I20240503 13:58:19.684402 13494 maintenance_manager.cc:419] P 7690ee79577c45e3a2c8f8b552c0bff8: Scheduling FlushMRSOp(7b57f64985b74e5fa75a5603ab8cc4bf): perf score=0.033343
I20240503 13:58:19.709206 13384 maintenance_manager.cc:643] P 7690ee79577c45e3a2c8f8b552c0bff8: FlushMRSOp(7b57f64985b74e5fa75a5603ab8cc4bf) complete. Timing: real 0.020s user 0.016s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":908,"dirs.run_cpu_time_us":178,"dirs.run_wall_time_us":4127,"drs_written":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1029,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":214,"threads_started":2,"wal-append.queue_time_us":182}
I20240503 13:58:19.711107 13494 maintenance_manager.cc:419] P 7690ee79577c45e3a2c8f8b552c0bff8: Scheduling UndoDeltaBlockGCOp(7b57f64985b74e5fa75a5603ab8cc4bf): 4583 bytes on disk
I20240503 13:58:19.711571 13384 maintenance_manager.cc:643] P 7690ee79577c45e3a2c8f8b552c0bff8: UndoDeltaBlockGCOp(7b57f64985b74e5fa75a5603ab8cc4bf) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4}
I20240503 13:58:19.713102 13101 maintenance_manager.cc:419] P 60ba3ffe3b7746469834cd91c84be14e: Scheduling FlushMRSOp(30d2b5ede30744cfb46ec1be232a1405): perf score=0.033354
W20240503 13:58:19.722294 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 13:58:19.730281 12991 maintenance_manager.cc:643] P 60ba3ffe3b7746469834cd91c84be14e: FlushMRSOp(30d2b5ede30744cfb46ec1be232a1405) complete. Timing: real 0.017s user 0.014s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":163,"dirs.run_cpu_time_us":163,"dirs.run_wall_time_us":1396,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":640,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":130,"threads_started":2,"wal-append.queue_time_us":153}
I20240503 13:58:19.732053 13101 maintenance_manager.cc:419] P 60ba3ffe3b7746469834cd91c84be14e: Scheduling FlushMRSOp(7b57f64985b74e5fa75a5603ab8cc4bf): perf score=0.033357
I20240503 13:58:19.741281 13232 maintenance_manager.cc:419] P 8ccf5a4ac450414c9d3579bd5800b770: Scheduling FlushMRSOp(30d2b5ede30744cfb46ec1be232a1405): perf score=0.033361
W20240503 13:58:19.749614 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 13:58:19.770435 12991 maintenance_manager.cc:643] P 60ba3ffe3b7746469834cd91c84be14e: FlushMRSOp(7b57f64985b74e5fa75a5603ab8cc4bf) complete. Timing: real 0.038s user 0.014s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":2085,"dirs.run_cpu_time_us":204,"dirs.run_wall_time_us":8081,"drs_written":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4,"lbm_write_time_us":644,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":65,"threads_started":1,"wal-append.queue_time_us":147}
I20240503 13:58:19.771320 13122 maintenance_manager.cc:643] P 8ccf5a4ac450414c9d3579bd5800b770: FlushMRSOp(30d2b5ede30744cfb46ec1be232a1405) complete. Timing: real 0.030s user 0.016s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":10005,"drs_written":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4,"lbm_write_time_us":634,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":204,"threads_started":2,"wal-append.queue_time_us":5116}
I20240503 13:58:19.772395 13101 maintenance_manager.cc:419] P 60ba3ffe3b7746469834cd91c84be14e: Scheduling UndoDeltaBlockGCOp(30d2b5ede30744cfb46ec1be232a1405): 4923 bytes on disk
I20240503 13:58:19.772877 12991 maintenance_manager.cc:643] P 60ba3ffe3b7746469834cd91c84be14e: UndoDeltaBlockGCOp(30d2b5ede30744cfb46ec1be232a1405) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
I20240503 13:58:19.773144 13232 maintenance_manager.cc:419] P 8ccf5a4ac450414c9d3579bd5800b770: Scheduling UndoDeltaBlockGCOp(30d2b5ede30744cfb46ec1be232a1405): 4923 bytes on disk
I20240503 13:58:19.773712 13122 maintenance_manager.cc:643] P 8ccf5a4ac450414c9d3579bd5800b770: UndoDeltaBlockGCOp(30d2b5ede30744cfb46ec1be232a1405) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
I20240503 13:58:19.774428 13101 maintenance_manager.cc:419] P 60ba3ffe3b7746469834cd91c84be14e: Scheduling UndoDeltaBlockGCOp(7b57f64985b74e5fa75a5603ab8cc4bf): 4583 bytes on disk
I20240503 13:58:19.774843 12991 maintenance_manager.cc:643] P 60ba3ffe3b7746469834cd91c84be14e: UndoDeltaBlockGCOp(7b57f64985b74e5fa75a5603ab8cc4bf) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4}
I20240503 13:58:19.775420 13232 maintenance_manager.cc:419] P 8ccf5a4ac450414c9d3579bd5800b770: Scheduling FlushMRSOp(258d1ec1f0694c15a4d41bd064f35b4e): perf score=0.033370
W20240503 13:58:19.779870 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 13:58:19.802706 13122 maintenance_manager.cc:643] P 8ccf5a4ac450414c9d3579bd5800b770: FlushMRSOp(258d1ec1f0694c15a4d41bd064f35b4e) complete. Timing: real 0.027s user 0.008s sys 0.008s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":37,"dirs.run_cpu_time_us":173,"dirs.run_wall_time_us":4477,"drs_written":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4,"lbm_write_time_us":540,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":103,"threads_started":1,"wal-append.queue_time_us":4067}
I20240503 13:58:19.806972 13232 maintenance_manager.cc:419] P 8ccf5a4ac450414c9d3579bd5800b770: Scheduling UndoDeltaBlockGCOp(258d1ec1f0694c15a4d41bd064f35b4e): 4838 bytes on disk
I20240503 13:58:19.808197 13122 maintenance_manager.cc:643] P 8ccf5a4ac450414c9d3579bd5800b770: UndoDeltaBlockGCOp(258d1ec1f0694c15a4d41bd064f35b4e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
I20240503 13:58:19.818555 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:19.821130 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:19.830847 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:19.840342 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:19.840587 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:19.850116 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:19.851073 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:58:19.853065 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 13:58:19.879547 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 13:58:19.894984 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 13:58:19.896817 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 13:58:19.898365 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 13:58:19.905092 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 13:58:20.009142 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 13:58:20.026448 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:20.094025 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 13:58:20.120450 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 13:58:20.130899 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 13:58:20.130926 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 13:58:20.136019 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 13:58:20.189209 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 13:58:20.273540 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:58:20.274317 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 13:58:20.281735 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:20.281889 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:20.281992 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:20.295337 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:20.296689 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:20.297881 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:58:20.313226 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 13:58:20.334407 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 13:58:20.387233 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 13:58:20.424146 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 13:58:20.473448 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 13:58:20.498026 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:20.546237 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 13:58:20.636358 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 13:58:20.662252 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 13:58:20.667632 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 13:58:20.688037 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 13:58:20.691124 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 13:58:20.696209 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:20.700577 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:20.700821 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:20.703310 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:20.703321 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:20.704640 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:20.708094 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 13:58:20.803064 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 13:58:20.821633 13232 maintenance_manager.cc:419] P 8ccf5a4ac450414c9d3579bd5800b770: Scheduling FlushMRSOp(b291e3dd6a7a454ca4d736eff8dd3a41): perf score=0.033338
I20240503 13:58:20.835776 13122 maintenance_manager.cc:643] P 8ccf5a4ac450414c9d3579bd5800b770: FlushMRSOp(b291e3dd6a7a454ca4d736eff8dd3a41) complete. Timing: real 0.014s user 0.012s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":159,"dirs.run_cpu_time_us":183,"dirs.run_wall_time_us":1579,"drs_written":1,"lbm_read_time_us":35,"lbm_reads_lt_1ms":4,"lbm_write_time_us":638,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":135,"threads_started":2,"wal-append.queue_time_us":131}
I20240503 13:58:20.837390 13232 maintenance_manager.cc:419] P 8ccf5a4ac450414c9d3579bd5800b770: Scheduling FlushMRSOp(b82134ddae61479eb2ced348d7558799): perf score=0.033339
W20240503 13:58:20.846572 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 13:58:20.850484 13122 maintenance_manager.cc:643] P 8ccf5a4ac450414c9d3579bd5800b770: FlushMRSOp(b82134ddae61479eb2ced348d7558799) complete. Timing: real 0.013s user 0.011s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":57,"dirs.run_cpu_time_us":175,"dirs.run_wall_time_us":1555,"drs_written":1,"lbm_read_time_us":40,"lbm_reads_lt_1ms":4,"lbm_write_time_us":440,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":67,"threads_started":1,"wal-append.queue_time_us":140}
I20240503 13:58:20.852430 13232 maintenance_manager.cc:419] P 8ccf5a4ac450414c9d3579bd5800b770: Scheduling UndoDeltaBlockGCOp(b82134ddae61479eb2ced348d7558799): 4923 bytes on disk
I20240503 13:58:20.852903 13122 maintenance_manager.cc:643] P 8ccf5a4ac450414c9d3579bd5800b770: UndoDeltaBlockGCOp(b82134ddae61479eb2ced348d7558799) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4}
I20240503 13:58:20.854627 13232 maintenance_manager.cc:419] P 8ccf5a4ac450414c9d3579bd5800b770: Scheduling UndoDeltaBlockGCOp(b291e3dd6a7a454ca4d736eff8dd3a41): 4581 bytes on disk
I20240503 13:58:20.855006 13122 maintenance_manager.cc:643] P 8ccf5a4ac450414c9d3579bd5800b770: UndoDeltaBlockGCOp(b291e3dd6a7a454ca4d736eff8dd3a41) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4}
W20240503 13:58:20.915189 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 13:58:20.933065 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 13:58:20.939122 13363 maintenance_manager.cc:419] P ba0194098aa84449ae17e8a0d4c43703: Scheduling FlushMRSOp(b82134ddae61479eb2ced348d7558799): perf score=0.033368
W20240503 13:58:20.941905 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 13:58:20.953071 13253 maintenance_manager.cc:643] P ba0194098aa84449ae17e8a0d4c43703: FlushMRSOp(b82134ddae61479eb2ced348d7558799) complete. Timing: real 0.014s user 0.013s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":2506,"drs_written":1,"lbm_read_time_us":34,"lbm_reads_lt_1ms":4,"lbm_write_time_us":565,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":157,"threads_started":2,"wal-append.queue_time_us":266}
I20240503 13:58:20.954610 13363 maintenance_manager.cc:419] P ba0194098aa84449ae17e8a0d4c43703: Scheduling UndoDeltaBlockGCOp(b82134ddae61479eb2ced348d7558799): 4923 bytes on disk
I20240503 13:58:20.955001 13253 maintenance_manager.cc:643] P ba0194098aa84449ae17e8a0d4c43703: UndoDeltaBlockGCOp(b82134ddae61479eb2ced348d7558799) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
I20240503 13:58:20.956331 13363 maintenance_manager.cc:419] P ba0194098aa84449ae17e8a0d4c43703: Scheduling FlushMRSOp(d50d05d432e84d5eb0e54d1a68cdf158): perf score=0.033372
W20240503 13:58:20.958778 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:58:20.969118 13253 maintenance_manager.cc:643] P ba0194098aa84449ae17e8a0d4c43703: FlushMRSOp(d50d05d432e84d5eb0e54d1a68cdf158) complete. Timing: real 0.013s user 0.012s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":40,"dirs.run_cpu_time_us":229,"dirs.run_wall_time_us":1464,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":515,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":58,"threads_started":1,"wal-append.queue_time_us":195}
I20240503 13:58:20.970916 13494 maintenance_manager.cc:419] P 7690ee79577c45e3a2c8f8b552c0bff8: Scheduling FlushMRSOp(b291e3dd6a7a454ca4d736eff8dd3a41): perf score=0.033378
I20240503 13:58:20.970964 13363 maintenance_manager.cc:419] P ba0194098aa84449ae17e8a0d4c43703: Scheduling UndoDeltaBlockGCOp(d50d05d432e84d5eb0e54d1a68cdf158): 4838 bytes on disk
I20240503 13:58:20.971343 13253 maintenance_manager.cc:643] P ba0194098aa84449ae17e8a0d4c43703: UndoDeltaBlockGCOp(d50d05d432e84d5eb0e54d1a68cdf158) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
I20240503 13:58:20.985841 13384 maintenance_manager.cc:643] P 7690ee79577c45e3a2c8f8b552c0bff8: FlushMRSOp(b291e3dd6a7a454ca4d736eff8dd3a41) complete. Timing: real 0.015s user 0.013s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":1542,"dirs.run_cpu_time_us":211,"dirs.run_wall_time_us":1688,"drs_written":1,"lbm_read_time_us":35,"lbm_reads_lt_1ms":4,"lbm_write_time_us":476,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":141,"threads_started":2,"wal-append.queue_time_us":141}
I20240503 13:58:20.987524 13494 maintenance_manager.cc:419] P 7690ee79577c45e3a2c8f8b552c0bff8: Scheduling UndoDeltaBlockGCOp(b291e3dd6a7a454ca4d736eff8dd3a41): 4581 bytes on disk
I20240503 13:58:20.987912 13384 maintenance_manager.cc:643] P 7690ee79577c45e3a2c8f8b552c0bff8: UndoDeltaBlockGCOp(b291e3dd6a7a454ca4d736eff8dd3a41) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
I20240503 13:58:21.035403 13101 maintenance_manager.cc:419] P 60ba3ffe3b7746469834cd91c84be14e: Scheduling FlushMRSOp(d50d05d432e84d5eb0e54d1a68cdf158): perf score=0.033393
I20240503 13:58:21.060678 12991 maintenance_manager.cc:643] P 60ba3ffe3b7746469834cd91c84be14e: FlushMRSOp(d50d05d432e84d5eb0e54d1a68cdf158) complete. Timing: real 0.025s user 0.016s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":11441,"drs_written":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4,"lbm_write_time_us":633,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":346,"threads_started":2,"wal-append.queue_time_us":7413}
I20240503 13:58:21.062618 13101 maintenance_manager.cc:419] P 60ba3ffe3b7746469834cd91c84be14e: Scheduling UndoDeltaBlockGCOp(d50d05d432e84d5eb0e54d1a68cdf158): 4838 bytes on disk
I20240503 13:58:21.063040 12991 maintenance_manager.cc:643] P 60ba3ffe3b7746469834cd91c84be14e: UndoDeltaBlockGCOp(d50d05d432e84d5eb0e54d1a68cdf158) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
W20240503 13:58:21.088824 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 13:58:21.111968 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 13:58:21.112764 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:21.114298 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:21.122833 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:58:21.123020 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 13:58:21.124315 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:21.124631 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:58:21.132853 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 13:58:21.149567 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:58:21.159945 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 13:58:21.176793 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:58:21.189620 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 13:58:21.274163 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 13:58:21.305550 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 13:58:21.403024 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 13:58:21.418644 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 13:58:21.418678 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 13:58:21.438584 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 13:58:21.495225 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:21.504390 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 13:58:21.542330 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 13:58:21.546027 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:21.550347 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:21.550719 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:21.552086 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:21.555725 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:21.576117 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:58:21.576891 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 13:58:21.590631 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:58:21.603379 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 13:58:21.710028 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 13:58:21.725414 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 13:58:21.747977 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 13:58:21.753914 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 13:58:21.761399 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 13:58:21.834798 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 13:58:21.855759 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 13:58:21.903653 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 13:58:21.933348 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 13:58:21.960932 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 13:58:21.979012 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:58:21.989343 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:21.991492 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:21.992123 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:22.000674 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:22.001499 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:22.003902 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:22.016661 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:58:22.076725 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 13:58:22.132036 15426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240503 13:58:22.137915 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 13:58:22.214968 13118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240503 13:58:22.215629 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 13:58:22.230024 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 13:58:22.231063 13363 maintenance_manager.cc:419] P ba0194098aa84449ae17e8a0d4c43703: Scheduling FlushMRSOp(66121c347e6c443dac7c5123826a7ae1): perf score=0.033362
I20240503 13:58:22.246043 13253 maintenance_manager.cc:643] P ba0194098aa84449ae17e8a0d4c43703: FlushMRSOp(66121c347e6c443dac7c5123826a7ae1) complete. Timing: real 0.015s user 0.013s sys 0.001s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":166,"dirs.run_cpu_time_us":167,"dirs.run_wall_time_us":1548,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":572,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":137,"threads_started":2,"wal-append.queue_time_us":125}
I20240503 13:58:22.247259 13494 maintenance_manager.cc:419] P 7690ee79577c45e3a2c8f8b552c0bff8: Scheduling FlushMRSOp(05917bf47aea42d397ccae57c3f29416): perf score=0.033365
I20240503 13:58:22.247952 13363 maintenance_manager.cc:419] P ba0194098aa84449ae17e8a0d4c43703: Scheduling FlushMRSOp(a746bcb2d718433f8c9acc8bd027a429): perf score=0.033365
W20240503 13:58:22.253333 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 13:58:22.267232 13384 maintenance_manager.cc:643] P 7690ee79577c45e3a2c8f8b552c0bff8: FlushMRSOp(05917bf47aea42d397ccae57c3f29416) complete. Timing: real 0.020s user 0.014s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":510,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":140,"threads_started":2,"wal-append.queue_time_us":130}
I20240503 13:58:22.267366 13253 maintenance_manager.cc:643] P ba0194098aa84449ae17e8a0d4c43703: FlushMRSOp(a746bcb2d718433f8c9acc8bd027a429) complete. Timing: real 0.019s user 0.010s sys 0.001s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":2725,"dirs.run_cpu_time_us":216,"dirs.run_wall_time_us":1513,"drs_written":1,"lbm_read_time_us":34,"lbm_reads_lt_1ms":4,"lbm_write_time_us":416,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":64,"threads_started":1,"wal-append.queue_time_us":4408}
I20240503 13:58:22.268853 13494 maintenance_manager.cc:419] P 7690ee79577c45e3a2c8f8b552c0bff8: Scheduling UndoDeltaBlockGCOp(05917bf47aea42d397ccae57c3f29416): 4583 bytes on disk
I20240503 13:58:22.269136 13363 maintenance_manager.cc:419] P ba0194098aa84449ae17e8a0d4c43703: Scheduling UndoDeltaBlockGCOp(66121c347e6c443dac7c5123826a7ae1): 4923 bytes on disk
I20240503 13:58:22.269237 13384 maintenance_manager.cc:643] P 7690ee79577c45e3a2c8f8b552c0bff8: UndoDeltaBlockGCOp(05917bf47aea42d397ccae57c3f29416) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
I20240503 13:58:22.269496 13253 maintenance_manager.cc:643] P ba0194098aa84449ae17e8a0d4c43703: UndoDeltaBlockGCOp(66121c347e6c443dac7c5123826a7ae1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4}
I20240503 13:58:22.270951 13363 maintenance_manager.cc:419] P ba0194098aa84449ae17e8a0d4c43703: Scheduling UndoDeltaBlockGCOp(a746bcb2d718433f8c9acc8bd027a429): 4838 bytes on disk
I20240503 13:58:22.271375 13253 maintenance_manager.cc:643] P ba0194098aa84449ae17e8a0d4c43703: UndoDeltaBlockGCOp(a746bcb2d718433f8c9acc8bd027a429) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
W20240503 13:58:22.274367 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 13:58:22.290644 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 13:58:22.299539 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 13:58:22.301889 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 13:58:22.316408 15691 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 111 similar messages]
W20240503 13:58:22.319659 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 13:58:22.322814 13101 maintenance_manager.cc:419] P 60ba3ffe3b7746469834cd91c84be14e: Scheduling FlushMRSOp(05917bf47aea42d397ccae57c3f29416): perf score=0.033386
I20240503 13:58:22.338912 12991 maintenance_manager.cc:643] P 60ba3ffe3b7746469834cd91c84be14e: FlushMRSOp(05917bf47aea42d397ccae57c3f29416) complete. Timing: real 0.016s user 0.014s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":4285,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":519,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":165,"threads_started":2,"wal-append.queue_time_us":298}
I20240503 13:58:22.341123 13101 maintenance_manager.cc:419] P 60ba3ffe3b7746469834cd91c84be14e: Scheduling UndoDeltaBlockGCOp(05917bf47aea42d397ccae57c3f29416): 4583 bytes on disk
I20240503 13:58:22.341527 12991 maintenance_manager.cc:643] P 60ba3ffe3b7746469834cd91c84be14e: UndoDeltaBlockGCOp(05917bf47aea42d397ccae57c3f29416) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4}
I20240503 13:58:22.366698 13232 maintenance_manager.cc:419] P 8ccf5a4ac450414c9d3579bd5800b770: Scheduling FlushMRSOp(66121c347e6c443dac7c5123826a7ae1): perf score=0.033400
W20240503 13:58:22.372146 13249 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 110 similar messages]
I20240503 13:58:22.399432 13122 maintenance_manager.cc:643] P 8ccf5a4ac450414c9d3579bd5800b770: FlushMRSOp(66121c347e6c443dac7c5123826a7ae1) complete. Timing: real 0.032s user 0.017s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":967,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":156,"threads_started":2,"wal-append.queue_time_us":156}
I20240503 13:58:22.401472 13232 maintenance_manager.cc:419] P 8ccf5a4ac450414c9d3579bd5800b770: Scheduling FlushMRSOp(a746bcb2d718433f8c9acc8bd027a429): perf score=0.033407
I20240503 13:58:22.420957 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:22.434063 13122 maintenance_manager.cc:643] P 8ccf5a4ac450414c9d3579bd5800b770: FlushMRSOp(a746bcb2d718433f8c9acc8bd027a429) complete. Timing: real 0.030s user 0.014s sys 0.003s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":3049,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":563,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":65,"threads_started":1,"wal-append.queue_time_us":150}
I20240503 13:58:22.436228 13232 maintenance_manager.cc:419] P 8ccf5a4ac450414c9d3579bd5800b770: Scheduling UndoDeltaBlockGCOp(66121c347e6c443dac7c5123826a7ae1): 4923 bytes on disk
I20240503 13:58:22.436867 13122 maintenance_manager.cc:643] P 8ccf5a4ac450414c9d3579bd5800b770: UndoDeltaBlockGCOp(66121c347e6c443dac7c5123826a7ae1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4}
I20240503 13:58:22.438302 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:22.438319 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:58:22.438529 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 13:58:22.440045 13232 maintenance_manager.cc:419] P 8ccf5a4ac450414c9d3579bd5800b770: Scheduling UndoDeltaBlockGCOp(a746bcb2d718433f8c9acc8bd027a429): 4838 bytes on disk
I20240503 13:58:22.441064 13122 maintenance_manager.cc:643] P 8ccf5a4ac450414c9d3579bd5800b770: UndoDeltaBlockGCOp(a746bcb2d718433f8c9acc8bd027a429) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4}
I20240503 13:58:22.449292 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:22.452190 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:22.468313 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:22.475891 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:58:22.492559 15558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240503 13:58:22.493237 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 13:58:22.507899 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 13:58:22.513248 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:22.532217 12987 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240503 13:58:22.532871 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 13:58:22.551430 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 13:58:22.557610 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 13:58:22.576718 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240503 13:58:22.588586 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 13:58:22.716441 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 13:58:22.742982 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 13:58:22.777596 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 13:58:22.803315 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 13:58:22.811177 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 13:58:22.828189 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 13:58:22.887696 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:22.904964 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:22.913901 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:22.923048 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:22.937666 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:22.946502 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:22.960582 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:58:22.986708 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 13:58:23.006667 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 13:58:23.034976 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 13:58:23.097942 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:23.117827 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 13:58:23.161634 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 13:58:23.292537 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 13:58:23.296108 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 13:58:23.313987 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 13:58:23.344136 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:23.351639 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:23.352286 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:58:23.362722 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 13:58:23.364401 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:23.364656 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:23.364727 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:58:23.384591 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 13:58:23.403916 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:58:23.412783 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 13:58:23.419903 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 13:58:23.438184 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 13:58:23.546403 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 13:58:23.572377 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 13:58:23.590688 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 13:58:23.600940 13101 maintenance_manager.cc:419] P 60ba3ffe3b7746469834cd91c84be14e: Scheduling FlushMRSOp(a6a2c05fe3574ab98fb755935b6a92e1): perf score=0.033341
W20240503 13:58:23.605067 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 13:58:23.615036 12991 maintenance_manager.cc:643] P 60ba3ffe3b7746469834cd91c84be14e: FlushMRSOp(a6a2c05fe3574ab98fb755935b6a92e1) complete. Timing: real 0.014s user 0.012s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":161,"dirs.run_cpu_time_us":222,"dirs.run_wall_time_us":1534,"drs_written":1,"lbm_read_time_us":35,"lbm_reads_lt_1ms":4,"lbm_write_time_us":513,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":141,"threads_started":2,"wal-append.queue_time_us":140}
I20240503 13:58:23.616716 13101 maintenance_manager.cc:419] P 60ba3ffe3b7746469834cd91c84be14e: Scheduling UndoDeltaBlockGCOp(a6a2c05fe3574ab98fb755935b6a92e1): 4838 bytes on disk
I20240503 13:58:23.617178 12991 maintenance_manager.cc:643] P 60ba3ffe3b7746469834cd91c84be14e: UndoDeltaBlockGCOp(a6a2c05fe3574ab98fb755935b6a92e1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
W20240503 13:58:23.659034 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 13:58:23.675943 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:23.676748 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 13:58:23.676772 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 13:58:23.701213 13232 maintenance_manager.cc:419] P 8ccf5a4ac450414c9d3579bd5800b770: Scheduling FlushMRSOp(a6a2c05fe3574ab98fb755935b6a92e1): perf score=0.033371
W20240503 13:58:23.714277 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 13:58:23.728106 13122 maintenance_manager.cc:643] P 8ccf5a4ac450414c9d3579bd5800b770: FlushMRSOp(a6a2c05fe3574ab98fb755935b6a92e1) complete. Timing: real 0.027s user 0.016s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":178,"dirs.run_cpu_time_us":175,"dirs.run_wall_time_us":4740,"drs_written":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4,"lbm_write_time_us":560,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":145,"threads_started":2,"wal-append.queue_time_us":137}
I20240503 13:58:23.729825 13232 maintenance_manager.cc:419] P 8ccf5a4ac450414c9d3579bd5800b770: Scheduling FlushMRSOp(b1395d66d8e04ecdafd7a30c50aa1c9a): perf score=0.033377
W20240503 13:58:23.737555 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 13:58:23.743300 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 13:58:23.752509 13122 maintenance_manager.cc:643] P 8ccf5a4ac450414c9d3579bd5800b770: FlushMRSOp(b1395d66d8e04ecdafd7a30c50aa1c9a) complete. Timing: real 0.022s user 0.017s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":3600,"dirs.run_cpu_time_us":170,"dirs.run_wall_time_us":1516,"drs_written":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4,"lbm_write_time_us":982,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":81,"threads_started":1,"wal-append.queue_time_us":506}
I20240503 13:58:23.754714 13232 maintenance_manager.cc:419] P 8ccf5a4ac450414c9d3579bd5800b770: Scheduling UndoDeltaBlockGCOp(a6a2c05fe3574ab98fb755935b6a92e1): 4838 bytes on disk
I20240503 13:58:23.755981 13122 maintenance_manager.cc:643] P 8ccf5a4ac450414c9d3579bd5800b770: UndoDeltaBlockGCOp(a6a2c05fe3574ab98fb755935b6a92e1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":118,"lbm_reads_lt_1ms":4}
I20240503 13:58:23.760656 13232 maintenance_manager.cc:419] P 8ccf5a4ac450414c9d3579bd5800b770: Scheduling UndoDeltaBlockGCOp(b1395d66d8e04ecdafd7a30c50aa1c9a): 4583 bytes on disk
I20240503 13:58:23.761206 13122 maintenance_manager.cc:643] P 8ccf5a4ac450414c9d3579bd5800b770: UndoDeltaBlockGCOp(b1395d66d8e04ecdafd7a30c50aa1c9a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4}
I20240503 13:58:23.766108 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:58:23.773577 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 13:58:23.780318 13494 maintenance_manager.cc:419] P 7690ee79577c45e3a2c8f8b552c0bff8: Scheduling FlushMRSOp(1b714fbd920a447f866b8d4547275675): perf score=0.033396
I20240503 13:58:23.782238 13363 maintenance_manager.cc:419] P ba0194098aa84449ae17e8a0d4c43703: Scheduling FlushMRSOp(1b714fbd920a447f866b8d4547275675): perf score=0.033397
I20240503 13:58:23.785969 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 13:58:23.788012 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 13:58:23.790917 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:23.791689 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:23.800251 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:23.801331 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:23.816514 13384 maintenance_manager.cc:643] P 7690ee79577c45e3a2c8f8b552c0bff8: FlushMRSOp(1b714fbd920a447f866b8d4547275675) complete. Timing: real 0.035s user 0.017s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":4723,"dirs.run_cpu_time_us":240,"dirs.run_wall_time_us":6057,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":572,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":125,"threads_started":2,"wal-append.queue_time_us":11775}
I20240503 13:58:23.818862 13494 maintenance_manager.cc:419] P 7690ee79577c45e3a2c8f8b552c0bff8: Scheduling FlushMRSOp(b1395d66d8e04ecdafd7a30c50aa1c9a): perf score=0.033405
I20240503 13:58:23.826012 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:23.843577 13253 maintenance_manager.cc:643] P ba0194098aa84449ae17e8a0d4c43703: FlushMRSOp(1b714fbd920a447f866b8d4547275675) complete. Timing: real 0.061s user 0.015s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":31974,"dirs.run_cpu_time_us":197,"dirs.run_wall_time_us":1556,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1620,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":454,"threads_started":2,"wal-append.queue_time_us":24383}
I20240503 13:58:23.852895 13363 maintenance_manager.cc:419] P ba0194098aa84449ae17e8a0d4c43703: Scheduling UndoDeltaBlockGCOp(1b714fbd920a447f866b8d4547275675): 4923 bytes on disk
I20240503 13:58:23.853483 13253 maintenance_manager.cc:643] P ba0194098aa84449ae17e8a0d4c43703: UndoDeltaBlockGCOp(1b714fbd920a447f866b8d4547275675) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
W20240503 13:58:23.856855 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 13:58:23.857987 13384 maintenance_manager.cc:643] P 7690ee79577c45e3a2c8f8b552c0bff8: FlushMRSOp(b1395d66d8e04ecdafd7a30c50aa1c9a) complete. Timing: real 0.038s user 0.015s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":40,"dirs.run_cpu_time_us":149,"dirs.run_wall_time_us":1587,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":521,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":73,"threads_started":1,"wal-append.queue_time_us":20447}
I20240503 13:58:23.859385 13494 maintenance_manager.cc:419] P 7690ee79577c45e3a2c8f8b552c0bff8: Scheduling UndoDeltaBlockGCOp(1b714fbd920a447f866b8d4547275675): 4923 bytes on disk
I20240503 13:58:23.862216 13384 maintenance_manager.cc:643] P 7690ee79577c45e3a2c8f8b552c0bff8: UndoDeltaBlockGCOp(1b714fbd920a447f866b8d4547275675) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":341,"lbm_reads_lt_1ms":4}
I20240503 13:58:23.863608 13494 maintenance_manager.cc:419] P 7690ee79577c45e3a2c8f8b552c0bff8: Scheduling UndoDeltaBlockGCOp(b1395d66d8e04ecdafd7a30c50aa1c9a): 4583 bytes on disk
I20240503 13:58:23.863984 13384 maintenance_manager.cc:643] P 7690ee79577c45e3a2c8f8b552c0bff8: UndoDeltaBlockGCOp(b1395d66d8e04ecdafd7a30c50aa1c9a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
W20240503 13:58:23.962888 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 13:58:24.091575 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 13:58:24.119568 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:24.136719 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 13:58:24.136963 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 13:58:24.150360 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 13:58:24.238982 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:24.247496 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:24.249565 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:24.250258 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:58:24.252209 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 13:58:24.253585 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:58:24.259714 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 13:58:24.270395 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:24.282060 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:58:24.295071 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 13:58:24.370946 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 13:58:24.373951 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 13:58:24.431923 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 13:58:24.522109 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 13:58:24.549011 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 13:58:24.611271 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:24.621610 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 13:58:24.653354 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:24.655225 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:24.655333 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:24.657902 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:24.675741 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:24.681087 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:24.684751 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:58:24.685142 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 13:58:24.693526 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 13:58:24.710026 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 13:58:24.794153 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 13:58:24.796173 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 13:58:24.861284 13363 maintenance_manager.cc:419] P ba0194098aa84449ae17e8a0d4c43703: Scheduling FlushMRSOp(4939335096b94444a61503841e2e81eb): perf score=0.033336
I20240503 13:58:24.872390 13494 maintenance_manager.cc:419] P 7690ee79577c45e3a2c8f8b552c0bff8: Scheduling FlushMRSOp(b34b432bb82e465f87a6a3b05d7adfd2): perf score=0.033337
I20240503 13:58:24.874086 13253 maintenance_manager.cc:643] P ba0194098aa84449ae17e8a0d4c43703: FlushMRSOp(4939335096b94444a61503841e2e81eb) complete. Timing: real 0.013s user 0.011s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":181,"dirs.run_cpu_time_us":176,"dirs.run_wall_time_us":1533,"drs_written":1,"lbm_read_time_us":40,"lbm_reads_lt_1ms":4,"lbm_write_time_us":427,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":136,"threads_started":2,"wal-append.queue_time_us":129}
I20240503 13:58:24.876158 13363 maintenance_manager.cc:419] P ba0194098aa84449ae17e8a0d4c43703: Scheduling FlushMRSOp(397f0e3e877d4acd9157bdf99bee6255): perf score=0.033336
I20240503 13:58:24.876309 13101 maintenance_manager.cc:419] P 60ba3ffe3b7746469834cd91c84be14e: Scheduling FlushMRSOp(b34b432bb82e465f87a6a3b05d7adfd2): perf score=0.033335
I20240503 13:58:24.901429 13384 maintenance_manager.cc:643] P 7690ee79577c45e3a2c8f8b552c0bff8: FlushMRSOp(b34b432bb82e465f87a6a3b05d7adfd2) complete. Timing: real 0.029s user 0.017s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":194,"dirs.run_cpu_time_us":157,"dirs.run_wall_time_us":1357,"drs_written":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4,"lbm_write_time_us":516,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":174,"threads_started":2,"wal-append.queue_time_us":166}
W20240503 13:58:24.902724 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 13:58:24.903066 13253 maintenance_manager.cc:643] P ba0194098aa84449ae17e8a0d4c43703: FlushMRSOp(397f0e3e877d4acd9157bdf99bee6255) complete. Timing: real 0.027s user 0.016s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":230,"dirs.run_cpu_time_us":173,"dirs.run_wall_time_us":1586,"drs_written":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4,"lbm_write_time_us":656,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":90,"threads_started":1,"wal-append.queue_time_us":170}
I20240503 13:58:24.903365 13494 maintenance_manager.cc:419] P 7690ee79577c45e3a2c8f8b552c0bff8: Scheduling UndoDeltaBlockGCOp(b34b432bb82e465f87a6a3b05d7adfd2): 4581 bytes on disk
I20240503 13:58:24.904542 13384 maintenance_manager.cc:643] P 7690ee79577c45e3a2c8f8b552c0bff8: UndoDeltaBlockGCOp(b34b432bb82e465f87a6a3b05d7adfd2) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
I20240503 13:58:24.904759 13363 maintenance_manager.cc:419] P ba0194098aa84449ae17e8a0d4c43703: Scheduling UndoDeltaBlockGCOp(4939335096b94444a61503841e2e81eb): 4923 bytes on disk
I20240503 13:58:24.905336 13253 maintenance_manager.cc:643] P ba0194098aa84449ae17e8a0d4c43703: UndoDeltaBlockGCOp(4939335096b94444a61503841e2e81eb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
I20240503 13:58:24.906189 13494 maintenance_manager.cc:419] P 7690ee79577c45e3a2c8f8b552c0bff8: Scheduling FlushMRSOp(397f0e3e877d4acd9157bdf99bee6255): perf score=0.033344
I20240503 13:58:24.907011 13363 maintenance_manager.cc:419] P ba0194098aa84449ae17e8a0d4c43703: Scheduling UndoDeltaBlockGCOp(397f0e3e877d4acd9157bdf99bee6255): 4838 bytes on disk
I20240503 13:58:24.907709 13253 maintenance_manager.cc:643] P ba0194098aa84449ae17e8a0d4c43703: UndoDeltaBlockGCOp(397f0e3e877d4acd9157bdf99bee6255) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
I20240503 13:58:24.910178 12991 maintenance_manager.cc:643] P 60ba3ffe3b7746469834cd91c84be14e: FlushMRSOp(b34b432bb82e465f87a6a3b05d7adfd2) complete. Timing: real 0.034s user 0.012s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":3609,"dirs.run_cpu_time_us":225,"dirs.run_wall_time_us":4654,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2864,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":181,"threads_started":2,"wal-append.queue_time_us":229}
I20240503 13:58:24.912137 13101 maintenance_manager.cc:419] P 60ba3ffe3b7746469834cd91c84be14e: Scheduling UndoDeltaBlockGCOp(b34b432bb82e465f87a6a3b05d7adfd2): 4581 bytes on disk
I20240503 13:58:24.912789 12991 maintenance_manager.cc:643] P 60ba3ffe3b7746469834cd91c84be14e: UndoDeltaBlockGCOp(b34b432bb82e465f87a6a3b05d7adfd2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4}
W20240503 13:58:24.914232 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 13:58:24.921867 13384 maintenance_manager.cc:643] P 7690ee79577c45e3a2c8f8b552c0bff8: FlushMRSOp(397f0e3e877d4acd9157bdf99bee6255) complete. Timing: real 0.015s user 0.014s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":63,"dirs.run_cpu_time_us":167,"dirs.run_wall_time_us":1660,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":800,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":87,"threads_started":1,"wal-append.queue_time_us":207}
I20240503 13:58:24.923307 13494 maintenance_manager.cc:419] P 7690ee79577c45e3a2c8f8b552c0bff8: Scheduling UndoDeltaBlockGCOp(397f0e3e877d4acd9157bdf99bee6255): 4838 bytes on disk
I20240503 13:58:24.923677 13384 maintenance_manager.cc:643] P 7690ee79577c45e3a2c8f8b552c0bff8: UndoDeltaBlockGCOp(397f0e3e877d4acd9157bdf99bee6255) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4}
W20240503 13:58:24.955312 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 13:58:24.962782 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 13:58:24.964589 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 13:58:24.994400 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:25.006385 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 13:58:25.007169 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 13:58:25.021139 13232 maintenance_manager.cc:419] P 8ccf5a4ac450414c9d3579bd5800b770: Scheduling FlushMRSOp(4939335096b94444a61503841e2e81eb): perf score=0.033378
W20240503 13:58:25.026186 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 13:58:25.029266 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 13:58:25.060432 13122 maintenance_manager.cc:643] P 8ccf5a4ac450414c9d3579bd5800b770: FlushMRSOp(4939335096b94444a61503841e2e81eb) complete. Timing: real 0.039s user 0.018s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":184,"dirs.run_cpu_time_us":200,"dirs.run_wall_time_us":3951,"drs_written":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4,"lbm_write_time_us":504,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":148,"threads_started":2,"wal-append.queue_time_us":231}
I20240503 13:58:25.062608 13232 maintenance_manager.cc:419] P 8ccf5a4ac450414c9d3579bd5800b770: Scheduling UndoDeltaBlockGCOp(4939335096b94444a61503841e2e81eb): 4923 bytes on disk
I20240503 13:58:25.064925 13122 maintenance_manager.cc:643] P 8ccf5a4ac450414c9d3579bd5800b770: UndoDeltaBlockGCOp(4939335096b94444a61503841e2e81eb) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4}
I20240503 13:58:25.081977 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:25.086702 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:25.088614 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:25.089831 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:25.094951 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:25.103996 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:25.108255 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:58:25.166950 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 13:58:25.171603 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 13:58:25.274497 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 13:58:25.306656 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 13:58:25.329339 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 13:58:25.391021 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 13:58:25.478273 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 13:58:25.488123 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 13:58:25.501937 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 13:58:25.515977 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:25.522856 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:25.528713 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:25.536693 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:25.537396 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:25.541546 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:58:25.545118 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 13:58:25.554868 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 13:58:25.558858 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:58:25.577448 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:25.595216 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 13:58:25.683116 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 13:58:25.707141 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 13:58:25.756546 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 13:58:25.766356 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 13:58:25.798491 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 13:58:25.816715 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 13:58:25.879304 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 13:58:25.917893 13363 maintenance_manager.cc:419] P ba0194098aa84449ae17e8a0d4c43703: Scheduling FlushMRSOp(3bb148fcf60f410089420a9499f08086): perf score=0.033335
I20240503 13:58:25.921988 13101 maintenance_manager.cc:419] P 60ba3ffe3b7746469834cd91c84be14e: Scheduling FlushMRSOp(023517b9d4ee43f497d98c4d7a6037df): perf score=0.033339
I20240503 13:58:25.931138 13494 maintenance_manager.cc:419] P 7690ee79577c45e3a2c8f8b552c0bff8: Scheduling FlushMRSOp(3bb148fcf60f410089420a9499f08086): perf score=0.033341
W20240503 13:58:25.955976 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 13:58:25.962700 13253 maintenance_manager.cc:643] P ba0194098aa84449ae17e8a0d4c43703: FlushMRSOp(3bb148fcf60f410089420a9499f08086) complete. Timing: real 0.045s user 0.018s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":13911,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":15808,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":168,"threads_started":2,"wal-append.queue_time_us":6197}
I20240503 13:58:25.964949 13363 maintenance_manager.cc:419] P ba0194098aa84449ae17e8a0d4c43703: Scheduling UndoDeltaBlockGCOp(3bb148fcf60f410089420a9499f08086): 4923 bytes on disk
I20240503 13:58:25.965386 13253 maintenance_manager.cc:643] P ba0194098aa84449ae17e8a0d4c43703: UndoDeltaBlockGCOp(3bb148fcf60f410089420a9499f08086) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
I20240503 13:58:25.965847 12991 maintenance_manager.cc:643] P 60ba3ffe3b7746469834cd91c84be14e: FlushMRSOp(023517b9d4ee43f497d98c4d7a6037df) complete. Timing: real 0.044s user 0.016s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":6130,"dirs.run_cpu_time_us":188,"dirs.run_wall_time_us":1304,"drs_written":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4,"lbm_write_time_us":624,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":167,"threads_started":2,"wal-append.queue_time_us":257}
I20240503 13:58:25.967191 13363 maintenance_manager.cc:419] P ba0194098aa84449ae17e8a0d4c43703: Scheduling FlushMRSOp(866723d26350414197ec27544277db9c): perf score=0.033347
I20240503 13:58:25.967650 13101 maintenance_manager.cc:419] P 60ba3ffe3b7746469834cd91c84be14e: Scheduling UndoDeltaBlockGCOp(023517b9d4ee43f497d98c4d7a6037df): 4838 bytes on disk
I20240503 13:58:25.969144 12991 maintenance_manager.cc:643] P 60ba3ffe3b7746469834cd91c84be14e: UndoDeltaBlockGCOp(023517b9d4ee43f497d98c4d7a6037df) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":556,"lbm_reads_lt_1ms":4}
I20240503 13:58:25.975140 13384 maintenance_manager.cc:643] P 7690ee79577c45e3a2c8f8b552c0bff8: FlushMRSOp(3bb148fcf60f410089420a9499f08086) complete. Timing: real 0.044s user 0.017s sys 0.001s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":4915,"drs_written":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4,"lbm_write_time_us":585,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":288,"threads_started":2,"wal-append.queue_time_us":9420}
I20240503 13:58:25.977170 13494 maintenance_manager.cc:419] P 7690ee79577c45e3a2c8f8b552c0bff8: Scheduling UndoDeltaBlockGCOp(3bb148fcf60f410089420a9499f08086): 4923 bytes on disk
I20240503 13:58:25.978907 13384 maintenance_manager.cc:643] P 7690ee79577c45e3a2c8f8b552c0bff8: UndoDeltaBlockGCOp(3bb148fcf60f410089420a9499f08086) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4}
I20240503 13:58:26.004173 13253 maintenance_manager.cc:643] P ba0194098aa84449ae17e8a0d4c43703: FlushMRSOp(866723d26350414197ec27544277db9c) complete. Timing: real 0.037s user 0.016s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":48,"dirs.run_cpu_time_us":187,"dirs.run_wall_time_us":1691,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":776,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":81,"threads_started":1,"wal-append.queue_time_us":190}
I20240503 13:58:26.007040 13363 maintenance_manager.cc:419] P ba0194098aa84449ae17e8a0d4c43703: Scheduling UndoDeltaBlockGCOp(866723d26350414197ec27544277db9c): 4583 bytes on disk
W20240503 13:58:26.008507 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 13:58:26.009415 13253 maintenance_manager.cc:643] P ba0194098aa84449ae17e8a0d4c43703: UndoDeltaBlockGCOp(866723d26350414197ec27544277db9c) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4}
I20240503 13:58:26.009644 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:26.011823 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:26.011942 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:26.021193 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:26.025574 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:58:26.027444 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 13:58:26.041663 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:58:26.044559 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 13:58:26.045965 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 13:58:26.054391 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:26.074263 13232 maintenance_manager.cc:419] P 8ccf5a4ac450414c9d3579bd5800b770: Scheduling FlushMRSOp(866723d26350414197ec27544277db9c): perf score=0.033376
W20240503 13:58:26.078915 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:58:26.089104 13122 maintenance_manager.cc:643] P 8ccf5a4ac450414c9d3579bd5800b770: FlushMRSOp(866723d26350414197ec27544277db9c) complete. Timing: real 0.015s user 0.014s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":169,"dirs.run_cpu_time_us":170,"dirs.run_wall_time_us":1770,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":571,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":152,"threads_started":2,"wal-append.queue_time_us":180}
I20240503 13:58:26.091245 13232 maintenance_manager.cc:419] P 8ccf5a4ac450414c9d3579bd5800b770: Scheduling FlushMRSOp(023517b9d4ee43f497d98c4d7a6037df): perf score=0.033379
W20240503 13:58:26.097062 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 13:58:26.107219 13122 maintenance_manager.cc:643] P 8ccf5a4ac450414c9d3579bd5800b770: FlushMRSOp(023517b9d4ee43f497d98c4d7a6037df) complete. Timing: real 0.016s user 0.014s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":60,"dirs.run_cpu_time_us":226,"dirs.run_wall_time_us":2232,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":509,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":92,"threads_started":1,"wal-append.queue_time_us":218}
I20240503 13:58:26.109396 13232 maintenance_manager.cc:419] P 8ccf5a4ac450414c9d3579bd5800b770: Scheduling UndoDeltaBlockGCOp(023517b9d4ee43f497d98c4d7a6037df): 4838 bytes on disk
I20240503 13:58:26.110002 13122 maintenance_manager.cc:643] P 8ccf5a4ac450414c9d3579bd5800b770: UndoDeltaBlockGCOp(023517b9d4ee43f497d98c4d7a6037df) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
I20240503 13:58:26.111758 13232 maintenance_manager.cc:419] P 8ccf5a4ac450414c9d3579bd5800b770: Scheduling UndoDeltaBlockGCOp(866723d26350414197ec27544277db9c): 4583 bytes on disk
I20240503 13:58:26.112150 13122 maintenance_manager.cc:643] P 8ccf5a4ac450414c9d3579bd5800b770: UndoDeltaBlockGCOp(866723d26350414197ec27544277db9c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4}
W20240503 13:58:26.137437 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 13:58:26.155670 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 13:58:26.186901 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 13:58:26.194916 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 13:58:26.201754 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 13:58:26.249752 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 13:58:26.260529 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 13:58:26.404381 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 13:58:26.462816 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:26.471455 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:26.472896 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:26.475278 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:26.484072 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:26.490978 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:26.496846 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:58:26.503139 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 13:58:26.519389 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 13:58:26.531894 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 13:58:26.542098 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:26.564740 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 13:58:26.616591 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 13:58:26.626729 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 13:58:26.628237 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 13:58:26.697549 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 13:58:26.876140 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 13:58:26.877197 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 13:58:26.893249 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 13:58:26.919384 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:26.929474 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:26.935827 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:26.937136 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:58:26.950882 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 13:58:26.952070 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:58:26.952844 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 13:58:26.960676 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:26.964915 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:26.979122 13101 maintenance_manager.cc:419] P 60ba3ffe3b7746469834cd91c84be14e: Scheduling FlushMRSOp(dc544319be0b484f8e9e5a3fbc40f530): perf score=0.033352
I20240503 13:58:26.987228 13494 maintenance_manager.cc:419] P 7690ee79577c45e3a2c8f8b552c0bff8: Scheduling FlushMRSOp(fd6a3d908840456f82c346c4bf1d8160): perf score=0.033348
W20240503 13:58:27.004591 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 13:58:27.018383 13363 maintenance_manager.cc:419] P ba0194098aa84449ae17e8a0d4c43703: Scheduling FlushMRSOp(dc544319be0b484f8e9e5a3fbc40f530): perf score=0.033365
I20240503 13:58:27.020464 12991 maintenance_manager.cc:643] P 60ba3ffe3b7746469834cd91c84be14e: FlushMRSOp(dc544319be0b484f8e9e5a3fbc40f530) complete. Timing: real 0.041s user 0.016s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":160,"dirs.run_cpu_time_us":233,"dirs.run_wall_time_us":1716,"drs_written":1,"lbm_read_time_us":245,"lbm_reads_lt_1ms":4,"lbm_write_time_us":607,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":156,"threads_started":2,"wal-append.queue_time_us":335}
W20240503 13:58:27.021188 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 13:58:27.021412 13384 maintenance_manager.cc:643] P 7690ee79577c45e3a2c8f8b552c0bff8: FlushMRSOp(fd6a3d908840456f82c346c4bf1d8160) complete. Timing: real 0.034s user 0.013s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":151,"dirs.run_cpu_time_us":206,"dirs.run_wall_time_us":2467,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":593,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":164,"threads_started":2,"wal-append.queue_time_us":13566}
I20240503 13:58:27.023274 13494 maintenance_manager.cc:419] P 7690ee79577c45e3a2c8f8b552c0bff8: Scheduling UndoDeltaBlockGCOp(fd6a3d908840456f82c346c4bf1d8160): 4583 bytes on disk
I20240503 13:58:27.023792 13101 maintenance_manager.cc:419] P 60ba3ffe3b7746469834cd91c84be14e: Scheduling UndoDeltaBlockGCOp(dc544319be0b484f8e9e5a3fbc40f530): 4923 bytes on disk
I20240503 13:58:27.023912 13384 maintenance_manager.cc:643] P 7690ee79577c45e3a2c8f8b552c0bff8: UndoDeltaBlockGCOp(fd6a3d908840456f82c346c4bf1d8160) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
I20240503 13:58:27.024197 12991 maintenance_manager.cc:643] P 60ba3ffe3b7746469834cd91c84be14e: UndoDeltaBlockGCOp(dc544319be0b484f8e9e5a3fbc40f530) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
W20240503 13:58:27.026121 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:58:27.028654 13101 maintenance_manager.cc:419] P 60ba3ffe3b7746469834cd91c84be14e: Scheduling FlushMRSOp(fd6a3d908840456f82c346c4bf1d8160): perf score=0.033362
I20240503 13:58:27.047340 13253 maintenance_manager.cc:643] P ba0194098aa84449ae17e8a0d4c43703: FlushMRSOp(dc544319be0b484f8e9e5a3fbc40f530) complete. Timing: real 0.029s user 0.015s sys 0.002s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":205,"dirs.run_cpu_time_us":234,"dirs.run_wall_time_us":1644,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":570,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":231,"threads_started":2,"wal-append.queue_time_us":167}
I20240503 13:58:27.049453 13363 maintenance_manager.cc:419] P ba0194098aa84449ae17e8a0d4c43703: Scheduling FlushMRSOp(84f1ab11e74045f8a9d7033c5ead0c14): perf score=0.033373
I20240503 13:58:27.049654 12991 maintenance_manager.cc:643] P 60ba3ffe3b7746469834cd91c84be14e: FlushMRSOp(fd6a3d908840456f82c346c4bf1d8160) complete. Timing: real 0.021s user 0.015s sys 0.002s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":37,"dirs.run_cpu_time_us":160,"dirs.run_wall_time_us":1278,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":671,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":74,"threads_started":1,"wal-append.queue_time_us":142}
I20240503 13:58:27.051518 13101 maintenance_manager.cc:419] P 60ba3ffe3b7746469834cd91c84be14e: Scheduling UndoDeltaBlockGCOp(fd6a3d908840456f82c346c4bf1d8160): 4583 bytes on disk
I20240503 13:58:27.051960 12991 maintenance_manager.cc:643] P 60ba3ffe3b7746469834cd91c84be14e: UndoDeltaBlockGCOp(fd6a3d908840456f82c346c4bf1d8160) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
I20240503 13:58:27.063971 13253 maintenance_manager.cc:643] P ba0194098aa84449ae17e8a0d4c43703: FlushMRSOp(84f1ab11e74045f8a9d7033c5ead0c14) complete. Timing: real 0.014s user 0.012s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":59,"dirs.run_cpu_time_us":209,"dirs.run_wall_time_us":1548,"drs_written":1,"lbm_read_time_us":33,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1620,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":71,"threads_started":1,"wal-append.queue_time_us":158}
I20240503 13:58:27.065538 13363 maintenance_manager.cc:419] P ba0194098aa84449ae17e8a0d4c43703: Scheduling UndoDeltaBlockGCOp(dc544319be0b484f8e9e5a3fbc40f530): 4923 bytes on disk
I20240503 13:58:27.065932 13253 maintenance_manager.cc:643] P ba0194098aa84449ae17e8a0d4c43703: UndoDeltaBlockGCOp(dc544319be0b484f8e9e5a3fbc40f530) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
I20240503 13:58:27.067219 13363 maintenance_manager.cc:419] P ba0194098aa84449ae17e8a0d4c43703: Scheduling UndoDeltaBlockGCOp(84f1ab11e74045f8a9d7033c5ead0c14): 4838 bytes on disk
I20240503 13:58:27.067514 13253 maintenance_manager.cc:643] P ba0194098aa84449ae17e8a0d4c43703: UndoDeltaBlockGCOp(84f1ab11e74045f8a9d7033c5ead0c14) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4}
I20240503 13:58:27.122218 13232 maintenance_manager.cc:419] P 8ccf5a4ac450414c9d3579bd5800b770: Scheduling FlushMRSOp(84f1ab11e74045f8a9d7033c5ead0c14): perf score=0.033388
I20240503 13:58:27.136767 13122 maintenance_manager.cc:643] P 8ccf5a4ac450414c9d3579bd5800b770: FlushMRSOp(84f1ab11e74045f8a9d7033c5ead0c14) complete. Timing: real 0.014s user 0.014s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":193,"dirs.run_cpu_time_us":195,"dirs.run_wall_time_us":2197,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":454,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":158,"threads_started":2,"wal-append.queue_time_us":161}
I20240503 13:58:27.138607 13232 maintenance_manager.cc:419] P 8ccf5a4ac450414c9d3579bd5800b770: Scheduling UndoDeltaBlockGCOp(84f1ab11e74045f8a9d7033c5ead0c14): 4838 bytes on disk
I20240503 13:58:27.139304 13122 maintenance_manager.cc:643] P 8ccf5a4ac450414c9d3579bd5800b770: UndoDeltaBlockGCOp(84f1ab11e74045f8a9d7033c5ead0c14) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":126,"lbm_reads_lt_1ms":4}
W20240503 13:58:27.175983 15426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 118 similar messages]
W20240503 13:58:27.176721 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 13:58:27.254070 13118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240503 13:58:27.254913 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 13:58:27.265009 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 13:58:27.286747 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 13:58:27.315924 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 13:58:27.375912 15691 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 108 similar messages]
I20240503 13:58:27.404897 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:27.410537 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:27.413923 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:27.414273 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:27.419325 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:27.421288 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:58:27.424511 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 13:58:27.441717 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 13:58:27.456149 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:58:27.462950 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 13:58:27.466728 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 13:58:27.509779 13249 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 111 similar messages]
W20240503 13:58:27.515399 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 13:58:27.520009 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 13:58:27.523087 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 13:58:27.546501 12987 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 67 similar messages]
W20240503 13:58:27.566289 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 13:58:27.578122 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 13:58:27.582242 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240503 13:58:27.582911 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:27.607406 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 13:58:27.669859 15558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240503 13:58:27.670954 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 13:58:27.677748 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 13:58:27.690161 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 13:58:27.731479 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 13:58:27.747591 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 13:58:27.852986 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:27.862270 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:27.864734 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:27.865816 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:27.879848 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:27.883641 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:27.916018 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:58:28.034974 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 13:58:28.070943 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 13:58:28.106972 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 13:58:28.109611 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 13:58:28.133123 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 13:58:28.157469 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:28.189754 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 13:58:28.207245 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 13:58:28.215548 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 13:58:28.238514 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 13:58:28.248679 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 13:58:28.305066 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 13:58:28.316937 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 13:58:28.317765 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:28.318629 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:58:28.326306 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 13:58:28.330960 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:28.341060 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:28.345223 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 13:58:28.345968 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 13:58:28.346729 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:58:28.373057 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 13:58:28.375792 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:58:28.392813 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 13:58:28.536108 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 13:58:28.581663 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 13:58:28.673274 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 13:58:28.690410 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 13:58:28.734961 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:28.735194 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 13:58:28.737177 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:28.745527 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:28.752130 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:28.752655 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:28.752727 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:28.773283 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:28.786571 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:58:28.796239 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 13:58:28.807654 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 13:58:28.817257 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 13:58:28.829169 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 13:58:28.843963 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 13:58:28.920876 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 13:58:28.934197 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 13:58:28.983867 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 13:58:29.051568 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 13:58:29.116724 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 13:58:29.161291 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 13:58:29.175835 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 13:58:29.180100 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:29.187000 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:58:29.188891 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 13:58:29.193369 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:29.197361 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:29.204727 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:58:29.208685 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 13:58:29.221933 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:29.229835 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:58:29.321866 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:29.348708 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 13:58:29.412760 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 13:58:29.420007 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 13:58:29.422446 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 13:58:29.512810 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 13:58:29.547844 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 13:58:29.593200 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 13:58:29.601168 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 13:58:29.632786 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 13:58:29.653939 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:29.655843 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:29.667236 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:29.674818 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:29.676240 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:29.701445 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:29.706104 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:58:29.709097 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:29.752432 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 13:58:29.774760 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 13:58:29.834348 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 13:58:29.844823 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 13:58:29.858237 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 13:58:29.858709 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 13:58:29.870373 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 13:58:29.894011 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 13:58:29.950416 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 13:58:29.954252 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 13:58:30.000023 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 13:58:30.008988 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 13:58:30.061561 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 13:58:30.080437 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:30.084955 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:30.090770 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:30.098273 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:30.104751 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:30.123184 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:30.123376 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:58:30.133785 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 13:58:30.140118 15540 maintenance_manager.cc:419] P 61b887f0d40d497badeca5604c1f0546: Scheduling FlushMRSOp(fd6a3d908840456f82c346c4bf1d8160): perf score=0.033340
W20240503 13:58:30.143178 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 13:58:30.187268 15430 maintenance_manager.cc:643] P 61b887f0d40d497badeca5604c1f0546: FlushMRSOp(fd6a3d908840456f82c346c4bf1d8160) complete. Timing: real 0.047s user 0.030s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":28556,"dirs.queue_time_us":8352,"dirs.run_cpu_time_us":192,"dirs.run_wall_time_us":989,"drs_written":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4,"lbm_write_time_us":623,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":227,"threads_started":3,"wal-append.queue_time_us":1371}
I20240503 13:58:30.189956 15540 maintenance_manager.cc:419] P 61b887f0d40d497badeca5604c1f0546: Scheduling FlushMRSOp(b291e3dd6a7a454ca4d736eff8dd3a41): perf score=0.033337
W20240503 13:58:30.198204 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 13:58:30.220649 15430 maintenance_manager.cc:643] P 61b887f0d40d497badeca5604c1f0546: FlushMRSOp(b291e3dd6a7a454ca4d736eff8dd3a41) complete. Timing: real 0.030s user 0.021s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":547,"dirs.run_cpu_time_us":179,"dirs.run_wall_time_us":1636,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":564,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":110,"threads_started":1,"wal-append.queue_time_us":266}
I20240503 13:58:30.223063 15540 maintenance_manager.cc:419] P 61b887f0d40d497badeca5604c1f0546: Scheduling UndoDeltaBlockGCOp(fd6a3d908840456f82c346c4bf1d8160): 4583 bytes on disk
I20240503 13:58:30.223707 15430 maintenance_manager.cc:643] P 61b887f0d40d497badeca5604c1f0546: UndoDeltaBlockGCOp(fd6a3d908840456f82c346c4bf1d8160) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
I20240503 13:58:30.226126 15540 maintenance_manager.cc:419] P 61b887f0d40d497badeca5604c1f0546: Scheduling UndoDeltaBlockGCOp(b291e3dd6a7a454ca4d736eff8dd3a41): 4581 bytes on disk
W20240503 13:58:30.226455 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 13:58:30.226583 15430 maintenance_manager.cc:643] P 61b887f0d40d497badeca5604c1f0546: UndoDeltaBlockGCOp(b291e3dd6a7a454ca4d736eff8dd3a41) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
W20240503 13:58:30.233976 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 13:58:30.258898 15805 maintenance_manager.cc:419] P 76c386bef61a4bd5a907266a5118ffe2: Scheduling FlushMRSOp(b47b28a4766a425e8b9bdfea882f979d): perf score=0.033358
I20240503 13:58:30.282486 15672 maintenance_manager.cc:419] P 009f849ee7b54c80a3f5264695047e4d: Scheduling FlushMRSOp(4647faf6a457415d8ae2b6691f452eab): perf score=0.033371
W20240503 13:58:30.299681 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:58:30.317498 15695 maintenance_manager.cc:643] P 76c386bef61a4bd5a907266a5118ffe2: FlushMRSOp(b47b28a4766a425e8b9bdfea882f979d) complete. Timing: real 0.058s user 0.034s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":193,"drs_written":1,"lbm_read_time_us":160,"lbm_reads_lt_1ms":4,"lbm_write_time_us":902,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":329,"threads_started":3,"wal-append.queue_time_us":222}
I20240503 13:58:30.320188 15805 maintenance_manager.cc:419] P 76c386bef61a4bd5a907266a5118ffe2: Scheduling UndoDeltaBlockGCOp(b47b28a4766a425e8b9bdfea882f979d): 4836 bytes on disk
I20240503 13:58:30.321075 15695 maintenance_manager.cc:643] P 76c386bef61a4bd5a907266a5118ffe2: UndoDeltaBlockGCOp(b47b28a4766a425e8b9bdfea882f979d) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4}
I20240503 13:58:30.368432 15562 maintenance_manager.cc:643] P 009f849ee7b54c80a3f5264695047e4d: FlushMRSOp(4647faf6a457415d8ae2b6691f452eab) complete. Timing: real 0.085s user 0.034s sys 0.003s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":178,"dirs.queue_time_us":11834,"dirs.run_cpu_time_us":189,"dirs.run_wall_time_us":1704,"drs_written":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1242,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":265,"threads_started":3,"wal-append.queue_time_us":4159}
I20240503 13:58:30.371596 15672 maintenance_manager.cc:419] P 009f849ee7b54c80a3f5264695047e4d: Scheduling UndoDeltaBlockGCOp(4647faf6a457415d8ae2b6691f452eab): 4838 bytes on disk
I20240503 13:58:30.374256 15562 maintenance_manager.cc:643] P 009f849ee7b54c80a3f5264695047e4d: UndoDeltaBlockGCOp(4647faf6a457415d8ae2b6691f452eab) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
W20240503 13:58:30.394475 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 13:58:30.427220 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 13:58:30.480338 15540 maintenance_manager.cc:419] P 61b887f0d40d497badeca5604c1f0546: Scheduling FlushMRSOp(9eb0d911701b4b2bb33a6b7ab59f08aa): perf score=0.033393
W20240503 13:58:30.482571 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 13:58:30.494635 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 13:58:30.512255 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 13:58:30.528143 15430 maintenance_manager.cc:643] P 61b887f0d40d497badeca5604c1f0546: FlushMRSOp(9eb0d911701b4b2bb33a6b7ab59f08aa) complete. Timing: real 0.048s user 0.017s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":32,"dirs.run_cpu_time_us":177,"dirs.run_wall_time_us":1430,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1064,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":129,"threads_started":1,"wal-append.queue_time_us":220}
I20240503 13:58:30.530431 15540 maintenance_manager.cc:419] P 61b887f0d40d497badeca5604c1f0546: Scheduling UndoDeltaBlockGCOp(9eb0d911701b4b2bb33a6b7ab59f08aa): 4583 bytes on disk
I20240503 13:58:30.531409 15430 maintenance_manager.cc:643] P 61b887f0d40d497badeca5604c1f0546: UndoDeltaBlockGCOp(9eb0d911701b4b2bb33a6b7ab59f08aa) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4}
I20240503 13:58:30.575044 15805 maintenance_manager.cc:419] P 76c386bef61a4bd5a907266a5118ffe2: Scheduling FlushMRSOp(1b714fbd920a447f866b8d4547275675): perf score=0.033381
W20240503 13:58:30.610153 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 13:58:30.624181 15695 maintenance_manager.cc:643] P 76c386bef61a4bd5a907266a5118ffe2: FlushMRSOp(1b714fbd920a447f866b8d4547275675) complete. Timing: real 0.049s user 0.020s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":42315,"compiler_manager_pool.run_cpu_time_us":16,"compiler_manager_pool.run_wall_time_us":16,"dirs.queue_time_us":9873,"dirs.run_cpu_time_us":204,"dirs.run_wall_time_us":1526,"drs_written":1,"lbm_read_time_us":39,"lbm_reads_lt_1ms":4,"lbm_write_time_us":601,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":81,"threads_started":1,"wal-append.queue_time_us":694}
I20240503 13:58:30.626492 15805 maintenance_manager.cc:419] P 76c386bef61a4bd5a907266a5118ffe2: Scheduling UndoDeltaBlockGCOp(1b714fbd920a447f866b8d4547275675): 4923 bytes on disk
I20240503 13:58:30.627342 15695 maintenance_manager.cc:643] P 76c386bef61a4bd5a907266a5118ffe2: UndoDeltaBlockGCOp(1b714fbd920a447f866b8d4547275675) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4}
I20240503 13:58:30.629154 15805 maintenance_manager.cc:419] P 76c386bef61a4bd5a907266a5118ffe2: Scheduling FlushMRSOp(05917bf47aea42d397ccae57c3f29416): perf score=0.033360
I20240503 13:58:30.670801 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:30.672971 15695 maintenance_manager.cc:643] P 76c386bef61a4bd5a907266a5118ffe2: FlushMRSOp(05917bf47aea42d397ccae57c3f29416) complete. Timing: real 0.035s user 0.016s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":42,"dirs.run_cpu_time_us":156,"dirs.run_wall_time_us":1389,"drs_written":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4,"lbm_write_time_us":988,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":86,"threads_started":1,"wal-append.queue_time_us":383}
I20240503 13:58:30.674985 15805 maintenance_manager.cc:419] P 76c386bef61a4bd5a907266a5118ffe2: Scheduling UndoDeltaBlockGCOp(05917bf47aea42d397ccae57c3f29416): 4583 bytes on disk
I20240503 13:58:30.678458 15695 maintenance_manager.cc:643] P 76c386bef61a4bd5a907266a5118ffe2: UndoDeltaBlockGCOp(05917bf47aea42d397ccae57c3f29416) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4}
I20240503 13:58:30.679744 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:30.684654 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:30.686228 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:30.687399 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:30.693007 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:30.712997 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:58:30.731240 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 13:58:30.738291 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 13:58:30.743450 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 13:58:30.769909 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 13:58:30.772271 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 13:58:30.781690 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 13:58:30.789495 15540 maintenance_manager.cc:419] P 61b887f0d40d497badeca5604c1f0546: Scheduling FlushMRSOp(3bb148fcf60f410089420a9499f08086): perf score=0.033384
I20240503 13:58:30.805938 15430 maintenance_manager.cc:643] P 61b887f0d40d497badeca5604c1f0546: FlushMRSOp(3bb148fcf60f410089420a9499f08086) complete. Timing: real 0.016s user 0.013s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":28,"dirs.run_cpu_time_us":175,"dirs.run_wall_time_us":1295,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":498,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":71,"threads_started":1,"wal-append.queue_time_us":149}
I20240503 13:58:30.807879 15540 maintenance_manager.cc:419] P 61b887f0d40d497badeca5604c1f0546: Scheduling UndoDeltaBlockGCOp(3bb148fcf60f410089420a9499f08086): 4923 bytes on disk
I20240503 13:58:30.808245 15430 maintenance_manager.cc:643] P 61b887f0d40d497badeca5604c1f0546: UndoDeltaBlockGCOp(3bb148fcf60f410089420a9499f08086) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4}
W20240503 13:58:30.830495 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 13:58:30.864459 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 13:58:30.880640 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:30.916527 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 13:58:30.953845 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 13:58:31.036514 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 13:58:31.071244 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 13:58:31.087519 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:31.088789 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:31.098345 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:31.100586 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:31.105998 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:31.109599 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:58:31.121146 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 13:58:31.132876 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:58:31.153822 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 13:58:31.207058 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 13:58:31.211576 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 13:58:31.243316 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 13:58:31.305225 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:31.323738 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 13:58:31.326542 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 13:58:31.346909 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 13:58:31.477442 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 13:58:31.494454 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 13:58:31.504292 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:31.504739 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:58:31.505403 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 13:58:31.508864 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:31.512059 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:31.516677 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:31.526185 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:31.555514 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:58:31.627359 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 13:58:31.628561 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 13:58:31.643543 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 13:58:31.658313 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 13:58:31.658779 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 13:58:31.667474 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 13:58:31.778940 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 13:58:31.794590 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 13:58:31.806241 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:31.880757 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 13:58:31.897531 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 13:58:31.900285 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 13:58:31.950659 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:31.954216 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:31.962647 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:31.962749 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:31.970960 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:31.977196 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:31.985030 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:58:32.036161 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 13:58:32.066115 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 13:58:32.071750 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 13:58:32.156162 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 13:58:32.166460 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 13:58:32.198947 15426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240503 13:58:32.308109 13118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240503 13:58:32.309051 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 13:58:32.325954 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 13:58:32.330073 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 13:58:32.380903 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 13:58:32.381363 15691 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 112 similar messages]
I20240503 13:58:32.382220 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:32.391710 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:32.395354 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:58:32.397338 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 13:58:32.400368 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:32.401603 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:32.415400 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:58:32.434238 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:32.445258 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 13:58:32.448715 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 13:58:32.448824 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 13:58:32.453246 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 13:58:32.477543 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 13:58:32.491104 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 13:58:32.545993 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 13:58:32.560189 12987 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240503 13:58:32.560882 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 13:58:32.575670 13249 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 114 similar messages]
W20240503 13:58:32.594146 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 13:58:32.607331 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 13:58:32.621188 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 13:58:32.715092 15558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240503 13:58:32.749226 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 13:58:32.813725 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:32.816380 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:32.818928 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:32.820721 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:32.826381 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:58:32.827405 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 73 similar messages]
I20240503 13:58:32.842842 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:32.858497 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:58:32.874963 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 13:58:32.901228 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 13:58:32.908241 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 13:58:32.909919 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:32.987115 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 13:58:33.115809 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 13:58:33.155612 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 13:58:33.208453 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 13:58:33.254200 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:33.256956 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:33.259016 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:33.261869 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:33.271826 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:33.282155 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:58:33.282896 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 13:58:33.284466 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:58:33.304723 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 13:58:33.342797 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 13:58:33.352752 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 13:58:33.400866 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 13:58:33.435881 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 13:58:33.484726 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 13:58:33.485057 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 13:58:33.518179 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:33.543941 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 13:58:33.567132 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 13:58:33.575783 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 13:58:33.615792 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 13:58:33.620894 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 13:58:33.627724 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 13:58:33.706439 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:33.707305 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:33.709759 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:58:33.710311 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 13:58:33.710826 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:33.713116 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:33.744639 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:33.747454 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:58:33.752655 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 13:58:33.776621 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 13:58:33.779654 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 13:58:33.822497 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 13:58:33.827643 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 13:58:33.846146 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 13:58:33.929863 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 13:58:33.942622 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 13:58:33.948791 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:33.966586 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 13:58:34.028769 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 13:58:34.040956 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 13:58:34.094239 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 13:58:34.127795 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 13:58:34.132536 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:34.133790 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:34.134496 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:34.147779 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:34.169965 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:34.170738 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:34.177954 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:58:34.211119 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 13:58:34.273962 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 13:58:34.309860 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 13:58:34.340394 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 13:58:34.396401 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 13:58:34.408546 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 13:58:34.427898 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 13:58:34.428349 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 13:58:34.520272 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 13:58:34.539914 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 13:58:34.560559 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:58:34.565972 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:34.566282 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:34.572106 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:34.580425 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:34.584003 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:34.594947 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:58:34.616542 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 13:58:34.621275 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:58:34.734297 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 13:58:34.866946 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 13:58:34.874449 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 13:58:34.874979 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 13:58:34.905892 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 13:58:34.913497 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 13:58:34.921895 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 13:58:34.931253 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 13:58:34.985150 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 13:58:34.985484 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 13:58:35.003590 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 13:58:35.010887 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 13:58:35.015269 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:35.015555 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:35.016400 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:35.021271 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:35.023861 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:35.045248 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:58:35.055271 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 13:58:35.082962 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:35.139961 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 13:58:35.174618 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 13:58:35.222813 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 13:58:35.295271 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 13:58:35.341634 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 13:58:35.342734 15540 maintenance_manager.cc:419] P 61b887f0d40d497badeca5604c1f0546: Scheduling FlushMRSOp(66121c347e6c443dac7c5123826a7ae1): perf score=0.033338
I20240503 13:58:35.362628 15430 maintenance_manager.cc:643] P 61b887f0d40d497badeca5604c1f0546: FlushMRSOp(66121c347e6c443dac7c5123826a7ae1) complete. Timing: real 0.020s user 0.017s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":3247,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":806,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":146,"threads_started":2,"wal-append.queue_time_us":223}
I20240503 13:58:35.364753 15540 maintenance_manager.cc:419] P 61b887f0d40d497badeca5604c1f0546: Scheduling UndoDeltaBlockGCOp(66121c347e6c443dac7c5123826a7ae1): 4923 bytes on disk
I20240503 13:58:35.365157 15430 maintenance_manager.cc:643] P 61b887f0d40d497badeca5604c1f0546: UndoDeltaBlockGCOp(66121c347e6c443dac7c5123826a7ae1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
I20240503 13:58:35.413887 15672 maintenance_manager.cc:419] P 009f849ee7b54c80a3f5264695047e4d: Scheduling FlushMRSOp(4939335096b94444a61503841e2e81eb): perf score=0.033340
I20240503 13:58:35.439797 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:35.446516 15562 maintenance_manager.cc:643] P 009f849ee7b54c80a3f5264695047e4d: FlushMRSOp(4939335096b94444a61503841e2e81eb) complete. Timing: real 0.032s user 0.017s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":400,"dirs.run_cpu_time_us":183,"dirs.run_wall_time_us":5832,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":626,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":148,"threads_started":2,"wal-append.queue_time_us":202}
I20240503 13:58:35.448808 15672 maintenance_manager.cc:419] P 009f849ee7b54c80a3f5264695047e4d: Scheduling UndoDeltaBlockGCOp(4939335096b94444a61503841e2e81eb): 4923 bytes on disk
I20240503 13:58:35.449658 15562 maintenance_manager.cc:643] P 009f849ee7b54c80a3f5264695047e4d: UndoDeltaBlockGCOp(4939335096b94444a61503841e2e81eb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4}
I20240503 13:58:35.451138 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:35.453861 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:35.454735 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:35.456022 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:35.469270 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:35.478137 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:58:35.483394 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 13:58:35.577448 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 13:58:35.586967 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:35.594298 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 13:58:35.601758 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 13:58:35.609925 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 13:58:35.653798 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 13:58:35.716696 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 13:58:35.732429 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 13:58:35.821235 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 13:58:35.865700 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:35.870615 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:58:35.874804 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 13:58:35.876708 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:35.884872 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:35.887467 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:35.893762 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:35.900601 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:58:35.985930 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 13:58:36.021622 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 13:58:36.022179 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 13:58:36.029971 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 13:58:36.045564 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 13:58:36.056396 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:36.102147 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 13:58:36.114370 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 13:58:36.115150 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 13:58:36.130327 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 13:58:36.164381 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 13:58:36.187777 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 13:58:36.223547 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 13:58:36.289383 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 13:58:36.312384 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:36.315351 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:36.316756 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:58:36.317379 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 13:58:36.320791 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:36.320895 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:36.332664 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:58:36.338359 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 13:58:36.352650 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 13:58:36.384088 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 13:58:36.441151 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 13:58:36.466586 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 13:58:36.491173 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 13:58:36.506649 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 13:58:36.517232 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:36.533671 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 13:58:36.586079 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 13:58:36.697765 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 13:58:36.741173 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:36.743366 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:36.752247 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:36.752861 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:36.753126 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:58:36.757861 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 13:58:36.759932 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:58:36.761346 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 13:58:36.778862 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:58:36.843015 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 13:58:36.850430 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 13:58:36.854562 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 13:58:36.856437 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 13:58:36.862969 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 13:58:36.900043 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 13:58:37.031917 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:37.032016 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 13:58:37.057582 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 13:58:37.157917 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:37.158600 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:58:37.162140 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 13:58:37.162751 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:37.169476 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:37.171816 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:37.184334 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:37.184991 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 13:58:37.199214 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 13:58:37.216574 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 13:58:37.217496 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 13:58:37.229580 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 13:58:37.251673 15426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240503 13:58:37.310889 13118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240503 13:58:37.324105 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 13:58:37.337399 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 13:58:37.403223 15691 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 107 similar messages]
W20240503 13:58:37.427804 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 13:58:37.509263 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 13:58:37.569499 12987 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 67 similar messages]
W20240503 13:58:37.569942 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 13:58:37.570030 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 13:58:37.595257 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:37.602226 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:58:37.602599 13249 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 110 similar messages]
W20240503 13:58:37.605235 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 13:58:37.606079 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 13:58:37.615590 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:58:37.619448 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 13:58:37.624248 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 13:58:37.627368 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:37.627516 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:37.628163 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:37.633422 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 13:58:37.648365 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:37.712641 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 13:58:37.730460 15805 maintenance_manager.cc:419] P 76c386bef61a4bd5a907266a5118ffe2: Scheduling FlushMRSOp(dc544319be0b484f8e9e5a3fbc40f530): perf score=0.033342
W20240503 13:58:37.730860 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 13:58:37.743257 15695 maintenance_manager.cc:643] P 76c386bef61a4bd5a907266a5118ffe2: FlushMRSOp(dc544319be0b484f8e9e5a3fbc40f530) complete. Timing: real 0.013s user 0.011s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":167,"dirs.run_cpu_time_us":312,"dirs.run_wall_time_us":1991,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":417,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":142,"threads_started":2,"wal-append.queue_time_us":167}
I20240503 13:58:37.744931 15805 maintenance_manager.cc:419] P 76c386bef61a4bd5a907266a5118ffe2: Scheduling UndoDeltaBlockGCOp(dc544319be0b484f8e9e5a3fbc40f530): 4923 bytes on disk
I20240503 13:58:37.745314 15695 maintenance_manager.cc:643] P 76c386bef61a4bd5a907266a5118ffe2: UndoDeltaBlockGCOp(dc544319be0b484f8e9e5a3fbc40f530) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4}
W20240503 13:58:37.787971 15558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240503 13:58:37.836656 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 13:58:37.843194 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240503 13:58:37.871889 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 13:58:37.915890 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 13:58:37.939198 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 13:58:37.998844 15805 maintenance_manager.cc:419] P 76c386bef61a4bd5a907266a5118ffe2: Scheduling FlushMRSOp(b82134ddae61479eb2ced348d7558799): perf score=0.033383
W20240503 13:58:38.009070 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 13:58:38.025619 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:38.038641 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:38.038884 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:38.039086 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:38.042822 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:38.045496 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:38.052038 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:58:38.066233 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:38.089327 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 13:58:38.094080 15695 maintenance_manager.cc:643] P 76c386bef61a4bd5a907266a5118ffe2: FlushMRSOp(b82134ddae61479eb2ced348d7558799) complete. Timing: real 0.095s user 0.017s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":200,"dirs.run_cpu_time_us":183,"dirs.run_wall_time_us":1847,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":782,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":163,"threads_started":1,"wal-append.queue_time_us":2769}
I20240503 13:58:38.095707 15805 maintenance_manager.cc:419] P 76c386bef61a4bd5a907266a5118ffe2: Scheduling UndoDeltaBlockGCOp(b82134ddae61479eb2ced348d7558799): 4923 bytes on disk
I20240503 13:58:38.096299 15695 maintenance_manager.cc:643] P 76c386bef61a4bd5a907266a5118ffe2: UndoDeltaBlockGCOp(b82134ddae61479eb2ced348d7558799) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4}
W20240503 13:58:38.098086 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 13:58:38.144634 15540 maintenance_manager.cc:419] P 61b887f0d40d497badeca5604c1f0546: Scheduling FlushMRSOp(a49b6f596d724b35b757ee69fe15e5a5): perf score=0.033394
I20240503 13:58:38.158663 15430 maintenance_manager.cc:643] P 61b887f0d40d497badeca5604c1f0546: FlushMRSOp(a49b6f596d724b35b757ee69fe15e5a5) complete. Timing: real 0.014s user 0.012s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":211,"dirs.run_cpu_time_us":221,"dirs.run_wall_time_us":1662,"drs_written":1,"lbm_read_time_us":38,"lbm_reads_lt_1ms":4,"lbm_write_time_us":538,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":152,"threads_started":2,"wal-append.queue_time_us":144}
I20240503 13:58:38.160156 15540 maintenance_manager.cc:419] P 61b887f0d40d497badeca5604c1f0546: Scheduling UndoDeltaBlockGCOp(a49b6f596d724b35b757ee69fe15e5a5): 4923 bytes on disk
I20240503 13:58:38.160583 15430 maintenance_manager.cc:643] P 61b887f0d40d497badeca5604c1f0546: UndoDeltaBlockGCOp(a49b6f596d724b35b757ee69fe15e5a5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
W20240503 13:58:38.242792 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 13:58:38.285653 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 13:58:38.325076 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 13:58:38.339682 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 13:58:38.372772 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 13:58:38.407948 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 13:58:38.448913 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:38.456189 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:38.457334 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:38.459129 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:38.464218 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:38.466377 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 13:58:38.474201 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 13:58:38.485495 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 13:58:38.507449 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:58:38.534775 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 13:58:38.541733 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 13:58:38.551465 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 13:58:38.568864 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 13:58:38.575202 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 13:58:38.593106 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 13:58:38.599483 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:38.625941 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 13:58:38.696812 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 13:58:38.698849 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 13:58:38.762899 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 13:58:38.794198 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 13:58:38.810878 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 13:58:38.817221 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 13:58:38.826159 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 13:58:38.887460 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:38.892009 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:38.897639 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 13:58:38.904614 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 13:58:38.908411 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:38.910362 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:38.936728 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:58:38.955539 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 13:58:38.966352 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:58:39.010565 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:39.065202 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 13:58:39.148351 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 13:58:39.149513 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 13:58:39.175273 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 13:58:39.208907 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 13:58:39.235275 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 13:58:39.290320 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 13:58:39.340728 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 13:58:39.347774 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 13:58:39.348861 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:39.349048 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:39.356158 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:39.358361 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:58:39.367177 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 13:58:39.368945 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:58:39.371934 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 13:58:39.374868 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 13:58:39.405205 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:58:39.410046 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 13:58:39.443554 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 13:58:39.493933 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 13:58:39.494055 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:39.551995 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 13:58:39.672111 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 13:58:39.744567 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 13:58:39.779274 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:39.780252 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:39.780903 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:39.790979 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:39.796128 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:39.796399 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:58:39.824600 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 13:58:39.825767 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:58:39.858600 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 13:58:39.870998 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:39.878937 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 13:58:39.910642 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 13:58:39.911547 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 13:58:39.916013 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 13:58:39.926352 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 13:58:40.039052 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 13:58:40.095404 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 13:58:40.146889 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 13:58:40.152033 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 13:58:40.204986 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:40.217259 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:40.217252 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:40.224915 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:58:40.225914 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 13:58:40.227336 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 13:58:40.233345 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:40.237042 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:40.237358 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:58:40.316071 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:40.416676 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 13:58:40.430009 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 13:58:40.451730 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 13:58:40.452770 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 13:58:40.476158 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 13:58:40.529559 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 13:58:40.576306 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 13:58:40.642297 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:40.649608 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:40.651144 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:40.651811 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:40.653164 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:40.658275 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:40.658545 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:58:40.662024 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 13:58:40.673203 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 13:58:40.705888 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 13:58:40.741899 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 13:58:40.759415 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 13:58:40.762095 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:40.802843 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 13:58:40.898293 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 13:58:40.903923 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 13:58:41.007761 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 13:58:41.064764 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:41.067474 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:41.069002 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:41.071980 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:41.080693 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:58:41.088027 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 13:58:41.096937 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:41.106117 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:58:41.122352 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 13:58:41.124362 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 13:58:41.149277 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 13:58:41.188205 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 13:58:41.218410 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 13:58:41.230111 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 13:58:41.248538 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 13:58:41.276985 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 13:58:41.287084 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 13:58:41.289881 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:41.291344 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 13:58:41.300532 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 13:58:41.305629 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 13:58:41.309774 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 13:58:41.435444 15540 maintenance_manager.cc:419] P 61b887f0d40d497badeca5604c1f0546: Scheduling FlushMRSOp(258d1ec1f0694c15a4d41bd064f35b4e): perf score=0.033399
W20240503 13:58:41.463500 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 13:58:41.468699 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 13:58:41.470074 15430 maintenance_manager.cc:643] P 61b887f0d40d497badeca5604c1f0546: FlushMRSOp(258d1ec1f0694c15a4d41bd064f35b4e) complete. Timing: real 0.034s user 0.013s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":2069,"dirs.run_cpu_time_us":176,"dirs.run_wall_time_us":4660,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":623,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":186,"threads_started":2,"wal-append.queue_time_us":241}
I20240503 13:58:41.472143 15540 maintenance_manager.cc:419] P 61b887f0d40d497badeca5604c1f0546: Scheduling UndoDeltaBlockGCOp(258d1ec1f0694c15a4d41bd064f35b4e): 4838 bytes on disk
I20240503 13:58:41.476711 15430 maintenance_manager.cc:643] P 61b887f0d40d497badeca5604c1f0546: UndoDeltaBlockGCOp(258d1ec1f0694c15a4d41bd064f35b4e) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":1222,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3}
I20240503 13:58:41.488631 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:41.490964 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:41.491017 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:41.491237 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:41.495064 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:41.498906 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:41.508658 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:41.521669 15672 maintenance_manager.cc:419] P 009f849ee7b54c80a3f5264695047e4d: Scheduling FlushMRSOp(b1395d66d8e04ecdafd7a30c50aa1c9a): perf score=0.033384
I20240503 13:58:41.545636 15562 maintenance_manager.cc:643] P 009f849ee7b54c80a3f5264695047e4d: FlushMRSOp(b1395d66d8e04ecdafd7a30c50aa1c9a) complete. Timing: real 0.024s user 0.016s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":2811,"dirs.run_cpu_time_us":171,"dirs.run_wall_time_us":2292,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":522,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":143,"threads_started":2,"wal-append.queue_time_us":583}
I20240503 13:58:41.547515 15672 maintenance_manager.cc:419] P 009f849ee7b54c80a3f5264695047e4d: Scheduling UndoDeltaBlockGCOp(b1395d66d8e04ecdafd7a30c50aa1c9a): 4583 bytes on disk
I20240503 13:58:41.547916 15562 maintenance_manager.cc:643] P 009f849ee7b54c80a3f5264695047e4d: UndoDeltaBlockGCOp(b1395d66d8e04ecdafd7a30c50aa1c9a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
W20240503 13:58:41.576121 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 13:58:41.584538 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 13:58:41.614077 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 13:58:41.621585 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 13:58:41.624589 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 13:58:41.725594 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 13:58:41.733899 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 13:58:41.765615 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:41.829195 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 13:58:41.835640 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 13:58:41.854491 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 13:58:41.868585 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 13:58:41.897461 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:41.899042 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:41.910802 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:41.914297 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:41.925490 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:41.932639 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:41.934171 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:58:41.938537 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 13:58:42.044975 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 13:58:42.098248 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 13:58:42.106456 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 13:58:42.129032 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 13:58:42.190330 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 13:58:42.260463 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:42.277546 15426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240503 13:58:42.314790 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 13:58:42.337903 13118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240503 13:58:42.340240 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 13:58:42.348953 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:42.350478 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:42.374436 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:42.374444 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:42.374950 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:42.379428 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:42.391817 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:58:42.464392 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 13:58:42.465703 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 13:58:42.473814 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 13:58:42.505573 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 13:58:42.522437 15691 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 110 similar messages]
W20240503 13:58:42.527626 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 13:58:42.530907 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 13:58:42.602090 12987 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240503 13:58:42.620927 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 13:58:42.622572 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 13:58:42.639653 13249 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 108 similar messages]
W20240503 13:58:42.729580 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 13:58:42.736771 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 13:58:42.758801 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 13:58:42.768054 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:42.771513 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:42.772651 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:42.773787 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:42.774238 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:42.782794 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:42.801487 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:58:42.826761 15558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240503 13:58:42.836369 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:42.877182 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 13:58:42.900436 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 13:58:42.928089 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 13:58:42.948184 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 13:58:42.952677 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240503 13:58:42.953316 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 13:58:43.008873 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 13:58:43.138235 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 13:58:43.173800 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:43.175995 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:43.180142 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:43.180369 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:58:43.187762 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 13:58:43.196987 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:58:43.201220 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 13:58:43.212595 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:43.214206 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:58:43.237406 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:43.271365 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 13:58:43.285667 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 13:58:43.288736 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 13:58:43.294597 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 13:58:43.324436 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 13:58:43.389535 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 13:58:43.514889 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 13:58:43.515879 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 13:58:43.545150 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 13:58:43.557328 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 13:58:43.595997 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:43.603232 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:43.608827 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:58:43.609107 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 13:58:43.609498 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:43.610693 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:43.615639 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:43.629065 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:58:43.635178 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 13:58:43.646544 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 13:58:43.675601 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 13:58:43.700335 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 13:58:43.719149 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:43.724397 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 13:58:43.754410 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 13:58:43.795686 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 13:58:43.839205 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 13:58:43.839588 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 13:58:43.911856 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 13:58:43.942152 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 13:58:43.960682 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 13:58:43.997840 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 13:58:44.015468 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 13:58:44.024821 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:44.032258 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:44.042698 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:44.045138 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:44.046909 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:44.052000 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:58:44.078359 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 13:58:44.095960 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:58:44.098546 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:44.127538 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 13:58:44.164451 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 13:58:44.183583 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 13:58:44.205402 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 13:58:44.256521 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 13:58:44.282469 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 13:58:44.305781 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 13:58:44.317974 15672 maintenance_manager.cc:419] P 009f849ee7b54c80a3f5264695047e4d: Scheduling FlushMRSOp(023517b9d4ee43f497d98c4d7a6037df): perf score=0.033387
I20240503 13:58:44.333195 15562 maintenance_manager.cc:643] P 009f849ee7b54c80a3f5264695047e4d: FlushMRSOp(023517b9d4ee43f497d98c4d7a6037df) complete. Timing: real 0.015s user 0.014s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":143,"dirs.run_cpu_time_us":171,"dirs.run_wall_time_us":1925,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":415,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":134,"threads_started":2,"wal-append.queue_time_us":134}
I20240503 13:58:44.335085 15672 maintenance_manager.cc:419] P 009f849ee7b54c80a3f5264695047e4d: Scheduling UndoDeltaBlockGCOp(023517b9d4ee43f497d98c4d7a6037df): 4838 bytes on disk
I20240503 13:58:44.335580 15562 maintenance_manager.cc:643] P 009f849ee7b54c80a3f5264695047e4d: UndoDeltaBlockGCOp(023517b9d4ee43f497d98c4d7a6037df) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
W20240503 13:58:44.347494 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 13:58:44.442044 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 13:58:44.472756 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:44.474289 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:44.479249 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:44.485467 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:44.488988 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:44.495519 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:44.495815 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:58:44.528064 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 13:58:44.535475 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 13:58:44.576646 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 13:58:44.633939 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 13:58:44.676589 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:44.728420 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 13:58:44.847934 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 13:58:44.888304 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:58:44.891438 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 13:58:44.896801 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:58:44.900115 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 13:58:44.903112 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:44.913374 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:44.917856 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:44.919073 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:44.925670 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 13:58:44.953430 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 13:58:44.959622 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 13:58:44.972920 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 13:58:44.994757 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 13:58:45.169719 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 13:58:45.170856 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 13:58:45.172470 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 13:58:45.177047 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 13:58:45.193498 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:45.301357 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 13:58:45.302729 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 13:58:45.323899 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:45.325548 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:45.328444 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:45.348696 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:45.354871 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:45.365753 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:58:45.386730 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 13:58:45.389760 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:58:45.390300 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 13:58:45.454983 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 13:58:45.534116 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 13:58:45.547125 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 13:58:45.559098 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 13:58:45.649572 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 13:58:45.657850 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 13:58:45.671399 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 13:58:45.754318 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 13:58:45.754861 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:45.760968 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 13:58:45.762245 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:45.767094 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:45.777493 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:45.779723 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:45.782930 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:45.783797 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:58:45.795542 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 13:58:45.807900 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 13:58:45.831395 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 13:58:45.892333 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 13:58:45.894237 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 13:58:45.904140 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 13:58:45.915117 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 13:58:45.999552 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 13:58:46.094290 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 13:58:46.127768 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 13:58:46.141595 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 13:58:46.165416 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 13:58:46.174947 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 13:58:46.182318 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 13:58:46.182701 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 13:58:46.182816 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:46.187901 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:46.188167 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:46.193959 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:46.198731 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:46.202970 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:58:46.228967 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 13:58:46.229591 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 13:58:46.237119 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:58:46.263465 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:46.284593 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 13:58:46.299930 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 13:58:46.437253 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 13:58:46.447569 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 13:58:46.452690 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 13:58:46.501559 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 13:58:46.506914 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 13:58:46.523715 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 13:58:46.554774 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 13:58:46.611198 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:46.616379 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:46.621336 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:46.628178 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:46.637183 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:46.653831 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:58:46.657449 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 13:58:46.666464 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 13:58:46.673594 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:58:46.747462 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:46.756006 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 13:58:46.870297 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 13:58:46.876353 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 13:58:46.996836 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 13:58:47.007786 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 13:58:47.032424 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 13:58:47.039614 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:47.043577 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:47.047755 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:58:47.053244 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 13:58:47.054039 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 13:58:47.066872 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:47.066929 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:47.067636 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:47.087028 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:58:47.181679 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 13:58:47.181764 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 13:58:47.223834 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 13:58:47.261405 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 13:58:47.263196 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:47.292846 15426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240503 13:58:47.342208 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 13:58:47.371466 13118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240503 13:58:47.373477 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 13:58:47.390467 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 13:58:47.446806 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 13:58:47.453342 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:47.461297 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:58:47.463088 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 13:58:47.473829 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 13:58:47.476435 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 13:58:47.477804 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:47.480710 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:47.482513 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:47.490995 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:47.492691 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:58:47.524097 15691 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 109 similar messages]
W20240503 13:58:47.585247 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 13:58:47.639544 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 13:58:47.648511 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 13:58:47.659826 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 13:58:47.681931 13249 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 110 similar messages]
W20240503 13:58:47.686051 12987 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240503 13:58:47.775317 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:47.779614 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 13:58:47.785446 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 13:58:47.811919 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 13:58:47.848995 15558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240503 13:58:47.865329 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 13:58:47.876636 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:47.876713 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:47.891275 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:47.895915 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:47.899541 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:47.903752 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:47.914661 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:58:47.924603 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 13:58:47.982905 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 13:58:47.988752 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240503 13:58:48.034601 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 13:58:48.042106 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 13:58:48.047567 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 13:58:48.171813 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 13:58:48.184303 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 13:58:48.197489 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:48.210934 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 13:58:48.306245 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:48.310106 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:58:48.311208 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 13:58:48.315091 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:48.315711 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:58:48.319378 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 13:58:48.320600 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:48.323149 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:48.328621 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:58:48.330561 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 13:58:48.346840 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 13:58:48.363689 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 13:58:48.420878 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 13:58:48.531060 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 13:58:48.540951 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 13:58:48.557807 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 13:58:48.643894 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 13:58:48.662536 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 13:58:48.700613 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 13:58:48.706969 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 13:58:48.717132 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:48.717414 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:48.723155 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:48.725406 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:58:48.733582 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:48.734870 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 13:58:48.736737 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:48.741638 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 13:58:48.742305 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 13:58:48.757450 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:58:48.800624 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 13:58:48.840745 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 13:58:48.841420 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 13:58:48.856184 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 13:58:48.880970 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 13:58:48.888521 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 13:58:48.895218 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 13:58:48.938755 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 13:58:49.041450 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 13:58:49.066049 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 13:58:49.067780 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 13:58:49.130676 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:49.132771 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:49.133507 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:49.140326 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:49.142761 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:49.143373 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:58:49.161600 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 13:58:49.176540 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:58:49.259961 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 13:58:49.267370 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 13:58:49.267513 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:49.358255 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 13:58:49.390774 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 13:58:49.411844 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 13:58:49.482270 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 13:58:49.545405 15540 maintenance_manager.cc:419] P 61b887f0d40d497badeca5604c1f0546: Scheduling FlushMRSOp(b34b432bb82e465f87a6a3b05d7adfd2): perf score=0.033400
I20240503 13:58:49.562422 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:49.563158 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:49.563308 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:49.563607 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:49.563997 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:49.571763 15430 maintenance_manager.cc:643] P 61b887f0d40d497badeca5604c1f0546: FlushMRSOp(b34b432bb82e465f87a6a3b05d7adfd2) complete. Timing: real 0.026s user 0.015s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":8157,"dirs.run_cpu_time_us":261,"dirs.run_wall_time_us":1370,"drs_written":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1827,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":136,"threads_started":2,"wal-append.queue_time_us":134}
I20240503 13:58:49.572533 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:49.573801 15540 maintenance_manager.cc:419] P 61b887f0d40d497badeca5604c1f0546: Scheduling UndoDeltaBlockGCOp(b34b432bb82e465f87a6a3b05d7adfd2): 4581 bytes on disk
I20240503 13:58:49.574322 15430 maintenance_manager.cc:643] P 61b887f0d40d497badeca5604c1f0546: UndoDeltaBlockGCOp(b34b432bb82e465f87a6a3b05d7adfd2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
W20240503 13:58:49.582772 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 13:58:49.583360 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:58:49.632655 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 13:58:49.663681 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 13:58:49.665344 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 13:58:49.692221 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 13:58:49.702867 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 13:58:49.773741 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:49.777470 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 13:58:49.869015 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 13:58:49.873095 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 13:58:49.885715 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 13:58:49.916807 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 13:58:49.921455 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 13:58:49.952140 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 13:58:49.994506 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:49.995379 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:49.998896 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:50.000372 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:50.004628 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:58:50.005039 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 13:58:50.005122 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 13:58:50.017341 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:50.030663 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:58:50.143085 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 13:58:50.236559 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 13:58:50.238950 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 13:58:50.272091 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 13:58:50.301820 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 13:58:50.333354 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 13:58:50.345582 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 13:58:50.357530 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 13:58:50.359234 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:50.378363 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 13:58:50.416344 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:50.418593 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:50.419565 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:50.427389 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:50.435197 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:50.435261 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:50.442380 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:58:50.465481 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 13:58:50.503644 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 13:58:50.582674 15540 maintenance_manager.cc:419] P 61b887f0d40d497badeca5604c1f0546: Scheduling FlushMRSOp(a6a2c05fe3574ab98fb755935b6a92e1): perf score=0.033374
W20240503 13:58:50.591113 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 13:58:50.599644 15430 maintenance_manager.cc:643] P 61b887f0d40d497badeca5604c1f0546: FlushMRSOp(a6a2c05fe3574ab98fb755935b6a92e1) complete. Timing: real 0.017s user 0.014s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":239,"dirs.run_cpu_time_us":176,"dirs.run_wall_time_us":1662,"drs_written":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4,"lbm_write_time_us":642,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":142,"threads_started":2,"wal-append.queue_time_us":151}
I20240503 13:58:50.601399 15540 maintenance_manager.cc:419] P 61b887f0d40d497badeca5604c1f0546: Scheduling UndoDeltaBlockGCOp(a6a2c05fe3574ab98fb755935b6a92e1): 4838 bytes on disk
I20240503 13:58:50.601761 15430 maintenance_manager.cc:643] P 61b887f0d40d497badeca5604c1f0546: UndoDeltaBlockGCOp(a6a2c05fe3574ab98fb755935b6a92e1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4}
W20240503 13:58:50.619186 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 13:58:50.627310 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 13:58:50.702032 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 13:58:50.702720 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 13:58:50.745782 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 13:58:50.746834 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 13:58:50.816424 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 13:58:50.820291 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 13:58:50.830857 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:50.835863 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:50.851830 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:50.852515 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:50.854601 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:50.855496 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:50.864682 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:58:50.866043 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:50.927090 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 13:58:50.992041 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 13:58:51.036695 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 13:58:51.042254 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 13:58:51.108351 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 13:58:51.140780 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 13:58:51.190186 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 13:58:51.248888 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 13:58:51.253511 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 13:58:51.260654 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:51.267169 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:51.269168 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:51.269883 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:51.270458 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:58:51.283215 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 13:58:51.286581 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 13:58:51.291306 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:58:51.313915 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 13:58:51.314716 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 13:58:51.320230 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 13:58:51.408066 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 13:58:51.408061 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 13:58:51.411747 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 13:58:51.434154 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:51.550144 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 13:58:51.607463 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 13:58:51.646944 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 13:58:51.663753 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 13:58:51.684639 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:51.687135 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:51.691283 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:51.695835 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:51.711161 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:51.715903 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:51.716465 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:58:51.717533 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 13:58:51.734071 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 13:58:51.764122 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 13:58:51.805478 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 13:58:51.822415 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:51.850732 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 13:58:51.952593 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 13:58:52.033458 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 13:58:52.091750 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:52.094610 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:52.096642 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:52.102996 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:58:52.110272 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 13:58:52.121863 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:52.125229 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 13:58:52.125365 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 13:58:52.145821 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:58:52.193825 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 13:58:52.203474 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 13:58:52.224256 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 13:58:52.255554 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 13:58:52.263765 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:52.266187 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 13:58:52.296306 15426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 118 similar messages]
W20240503 13:58:52.324234 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 13:58:52.395728 13118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240503 13:58:52.397176 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 13:58:52.429219 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 13:58:52.451779 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 13:58:52.475284 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 13:58:52.516218 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:52.517465 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:52.521687 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:52.524636 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:52.524920 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:52.529342 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 13:58:52.532662 15691 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 110 similar messages]
W20240503 13:58:52.533679 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 13:58:52.536986 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 13:58:52.537881 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:58:52.560056 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 13:58:52.622496 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 13:58:52.643976 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 13:58:52.651856 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:52.688761 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 13:58:52.692858 13249 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 111 similar messages]
W20240503 13:58:52.748092 12987 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240503 13:58:52.823381 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 13:58:52.871383 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 13:58:52.876878 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 13:58:52.882295 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 13:58:52.923424 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:52.926440 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:52.926707 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:52.933511 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:52.933923 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:58:52.938195 15558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 92 similar messages]
I20240503 13:58:52.940817 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:52.951435 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:58:52.960589 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 13:58:52.974915 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 13:58:52.995990 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240503 13:58:53.065215 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 13:58:53.081852 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 13:58:53.109856 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 13:58:53.113406 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:53.140012 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 13:58:53.148454 15672 maintenance_manager.cc:419] P 009f849ee7b54c80a3f5264695047e4d: Scheduling FlushMRSOp(7239f94a7c88433aa6062390a54e7562): perf score=0.033400
W20240503 13:58:53.161062 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 13:58:53.161538 15562 maintenance_manager.cc:643] P 009f849ee7b54c80a3f5264695047e4d: FlushMRSOp(7239f94a7c88433aa6062390a54e7562) complete. Timing: real 0.013s user 0.012s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":154,"dirs.run_cpu_time_us":164,"dirs.run_wall_time_us":1376,"drs_written":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4,"lbm_write_time_us":487,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":138,"threads_started":2,"wal-append.queue_time_us":130}
I20240503 13:58:53.163086 15672 maintenance_manager.cc:419] P 009f849ee7b54c80a3f5264695047e4d: Scheduling UndoDeltaBlockGCOp(7239f94a7c88433aa6062390a54e7562): 4923 bytes on disk
I20240503 13:58:53.163450 15562 maintenance_manager.cc:643] P 009f849ee7b54c80a3f5264695047e4d: UndoDeltaBlockGCOp(7239f94a7c88433aa6062390a54e7562) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
W20240503 13:58:53.215569 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 13:58:53.298857 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 13:58:53.307533 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 13:58:53.313150 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 13:58:53.344362 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:53.346925 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:53.347184 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:53.348970 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:53.368613 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:53.368657 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:58:53.382262 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 13:58:53.392566 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:58:53.440065 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 13:58:53.539276 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 13:58:53.569979 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 13:58:53.572351 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 13:58:53.579403 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 13:58:53.583501 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 13:58:53.595176 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 13:58:53.634048 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 13:58:53.642271 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:53.653124 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 13:58:53.680552 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 13:58:53.724195 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 13:58:53.764592 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:53.767715 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:58:53.769912 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 13:58:53.773020 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:53.775949 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:53.795131 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:53.795773 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:53.812530 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 13:58:53.827628 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 13:58:53.831609 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 13:58:53.889837 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 13:58:53.939424 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 13:58:53.939863 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 13:58:53.975522 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 13:58:53.978917 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 13:58:53.993918 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 13:58:54.110039 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 13:58:54.188110 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 13:58:54.189002 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 13:58:54.190732 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 13:58:54.200598 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:54.202913 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:54.204016 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:54.211346 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:54.233531 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:58:54.236231 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 13:58:54.242743 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 13:58:54.253541 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:58:54.256603 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:54.375275 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 13:58:54.425653 15672 maintenance_manager.cc:419] P 009f849ee7b54c80a3f5264695047e4d: Scheduling FlushMRSOp(30d2b5ede30744cfb46ec1be232a1405): perf score=0.033368
I20240503 13:58:54.438550 15562 maintenance_manager.cc:643] P 009f849ee7b54c80a3f5264695047e4d: FlushMRSOp(30d2b5ede30744cfb46ec1be232a1405) complete. Timing: real 0.013s user 0.011s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":153,"dirs.run_cpu_time_us":189,"dirs.run_wall_time_us":1527,"drs_written":1,"lbm_read_time_us":35,"lbm_reads_lt_1ms":4,"lbm_write_time_us":490,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":129,"threads_started":2,"wal-append.queue_time_us":111}
I20240503 13:58:54.439949 15672 maintenance_manager.cc:419] P 009f849ee7b54c80a3f5264695047e4d: Scheduling UndoDeltaBlockGCOp(30d2b5ede30744cfb46ec1be232a1405): 4923 bytes on disk
I20240503 13:58:54.440302 15562 maintenance_manager.cc:643] P 009f849ee7b54c80a3f5264695047e4d: UndoDeltaBlockGCOp(30d2b5ede30744cfb46ec1be232a1405) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4}
W20240503 13:58:54.442595 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 13:58:54.459412 15805 maintenance_manager.cc:419] P 76c386bef61a4bd5a907266a5118ffe2: Scheduling FlushMRSOp(df28831df92d4c12b63b2f7da6c6449a): perf score=0.033384
I20240503 13:58:54.472522 15695 maintenance_manager.cc:643] P 76c386bef61a4bd5a907266a5118ffe2: FlushMRSOp(df28831df92d4c12b63b2f7da6c6449a) complete. Timing: real 0.013s user 0.012s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":158,"dirs.run_cpu_time_us":197,"dirs.run_wall_time_us":1411,"drs_written":1,"lbm_read_time_us":33,"lbm_reads_lt_1ms":4,"lbm_write_time_us":547,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":138,"threads_started":2,"wal-append.queue_time_us":165}
I20240503 13:58:54.474017 15805 maintenance_manager.cc:419] P 76c386bef61a4bd5a907266a5118ffe2: Scheduling UndoDeltaBlockGCOp(df28831df92d4c12b63b2f7da6c6449a): 4838 bytes on disk
I20240503 13:58:54.474366 15695 maintenance_manager.cc:643] P 76c386bef61a4bd5a907266a5118ffe2: UndoDeltaBlockGCOp(df28831df92d4c12b63b2f7da6c6449a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4}
W20240503 13:58:54.514611 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 13:58:54.535532 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 13:58:54.544195 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 13:58:54.548285 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 13:58:54.633798 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:54.637823 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:54.640107 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:58:54.641098 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 13:58:54.642606 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:54.643072 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:58:54.646961 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 13:58:54.653065 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:54.665323 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:58:54.665849 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 13:58:54.729365 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 13:58:54.740947 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:54.832378 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 13:58:54.896826 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 13:58:54.899102 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 13:58:54.913530 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 13:58:54.985819 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 13:58:55.040594 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 13:58:55.052949 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:55.053649 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:55.059752 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 13:58:55.063344 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 13:58:55.069447 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:55.070187 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:55.072371 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:55.116845 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:58:55.156666 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 13:58:55.184114 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 13:58:55.237668 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 13:58:55.239300 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:55.262032 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 13:58:55.318807 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 13:58:55.364039 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 13:58:55.375955 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 13:58:55.410118 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 13:58:55.432940 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 13:58:55.477967 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:55.480880 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:55.483042 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:55.483837 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:55.493616 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:55.501966 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:55.504885 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 13:58:55.518070 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 13:58:55.537457 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 13:58:55.544605 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 13:58:55.661223 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 13:58:55.675124 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 13:58:55.711035 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:55.731817 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 13:58:55.734908 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 13:58:55.734932 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 13:58:55.751724 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 13:58:55.788638 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 13:58:55.815032 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 13:58:55.819437 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 13:58:55.820475 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 13:58:55.823863 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 13:58:55.840405 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 13:58:55.873898 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 13:58:55.903170 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:55.908226 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:55.912711 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:55.919588 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:58:55.928073 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 13:58:55.938707 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:55.945340 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:55.947731 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:58:56.072919 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 13:58:56.115187 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 13:58:56.143019 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 13:58:56.151949 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 13:58:56.169420 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:56.179575 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 13:58:56.275338 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 13:58:56.277395 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 13:58:56.318753 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 13:58:56.336814 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:56.341183 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:56.351346 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:56.361683 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:58:56.365795 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 13:58:56.367270 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:58:56.371477 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 13:58:56.372510 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:56.382335 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:58:56.511557 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 13:58:56.541092 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 13:58:56.561443 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 13:58:56.581034 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 13:58:56.586845 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:56.628001 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 13:58:56.651696 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 13:58:56.683578 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 13:58:56.703218 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 13:58:56.763171 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:56.763911 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:56.764374 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:58:56.784389 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 13:58:56.786181 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:56.786412 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:56.789829 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:56.807104 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:58:56.862706 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 13:58:56.897859 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 13:58:56.918068 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 13:58:56.933802 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 13:58:57.010942 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 13:58:57.015775 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 13:58:57.017522 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 13:58:57.175465 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:57.182142 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:57.192752 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:57.193429 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:57.204248 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:57.205451 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:57.216673 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:58:57.216920 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:57.228320 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 13:58:57.274942 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 13:58:57.305693 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 13:58:57.312536 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 13:58:57.313891 15426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 125 similar messages]
W20240503 13:58:57.322930 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 13:58:57.353192 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 13:58:57.428651 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 13:58:57.517488 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 13:58:57.522485 13118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240503 13:58:57.530953 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 13:58:57.551937 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 13:58:57.588768 15691 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 112 similar messages]
I20240503 13:58:57.610813 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:57.626015 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:57.626869 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:57.628592 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:57.629091 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:58:57.641597 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:58:57.646030 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:58:57.657562 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 13:58:57.657922 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:58:57.658239 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 13:58:57.672129 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 13:58:57.700438 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 13:58:57.708757 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 13:58:57.759779 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 13:58:57.780998 13249 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 109 similar messages]
W20240503 13:58:57.852403 12987 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240503 13:58:57.862857 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 13:58:57.875622 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 13:58:57.914742 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 13:58:58.035950 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:58.040616 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:58.040927 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:58.041841 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 13:58:58.044826 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240503 13:58:58.048542 15558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 91 similar messages]
I20240503 13:58:58.048645 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:58:58.058998 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:58:58.060925 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:58:58.066147 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 13:58:58.066556 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:58:58.115625 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 13:58:58.124907 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 13:58:58.128777 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 13:58:58.155864 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 13:58:58.160909 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 13:58:58.165762 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 13:58:58.168859 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 13:58:58.214982 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 13:58:58.294925 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 13:58:58.325554 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 13:58:58.346343 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 13:58:58.430056 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 13:58:58.471391 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:58:58.471995 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 13:58:58.474875 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:58.475157 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:58.486109 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:58.492650 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:58:58.497614 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 13:58:58.509620 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:58.525599 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:58:58.534113 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 13:58:58.596963 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 13:58:58.599756 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:58.602337 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 13:58:58.646320 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 13:58:58.650024 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 13:58:58.690433 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 13:58:58.700587 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 13:58:58.722716 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 13:58:58.766714 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 13:58:58.779719 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 13:58:58.812613 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 13:58:58.820276 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 13:58:58.890797 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:58.896036 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:58.900691 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:58.905159 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:58.910372 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:58.920640 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:58.920688 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:58:59.008989 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:59.022759 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 13:58:59.025290 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 13:58:59.040139 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 13:58:59.042204 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 13:58:59.045631 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 13:58:59.058573 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 13:58:59.178256 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 13:58:59.179220 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 13:58:59.199517 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 13:58:59.216279 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 13:58:59.321578 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:58:59.323765 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:58:59.325270 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:58:59.326292 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 13:58:59.328023 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:59.330192 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:59.332475 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:59.332798 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:58:59.502712 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 13:58:59.518273 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 13:58:59.548758 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 13:58:59.580540 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:58:59.581605 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 13:58:59.626454 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 13:58:59.646116 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 13:58:59.707521 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 13:58:59.727404 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:58:59.730374 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 13:58:59.732697 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:58:59.736841 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:58:59.738855 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:58:59.743240 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:58:59.746969 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 13:58:59.773758 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 13:58:59.774384 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 13:58:59.776397 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:58:59.806044 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 13:58:59.809073 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 13:58:59.816970 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 13:58:59.847357 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 13:58:59.913046 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 13:59:00.023125 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 13:59:00.050933 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 13:59:00.138792 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:59:00.149843 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:00.150977 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:59:00.153566 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 13:59:00.163666 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:00.174757 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:00.176174 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:59:00.186146 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:00.196805 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:59:00.210050 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 13:59:00.284834 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 13:59:00.295501 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 13:59:00.319983 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 13:59:00.362460 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 13:59:00.510735 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 13:59:00.534322 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 13:59:00.535490 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 13:59:00.555975 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 13:59:00.572644 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:00.578347 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:00.578591 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:00.584355 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:59:00.601384 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:00.604247 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:00.615612 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:59:00.658450 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 13:59:00.659307 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 13:59:00.676438 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 13:59:00.685725 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 13:59:00.702008 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 13:59:00.714175 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 13:59:00.716091 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:00.818465 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 13:59:00.853224 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 13:59:00.868957 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 13:59:00.931525 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 13:59:00.962817 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 13:59:00.990845 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:00.991112 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:00.994294 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:01.013725 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:01.016728 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:01.021412 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:59:01.025789 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:59:01.047500 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 13:59:01.070135 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 13:59:01.126225 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 13:59:01.163292 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 13:59:01.181129 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 13:59:01.195526 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 13:59:01.205667 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 13:59:01.221184 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 13:59:01.229923 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 13:59:01.240674 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 13:59:01.242408 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 13:59:01.257753 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:01.363631 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 13:59:01.411482 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:01.411810 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:01.416258 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:01.421851 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:59:01.422204 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:01.422988 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:01.432514 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:59:01.443800 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 13:59:01.465639 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 13:59:01.474309 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 13:59:01.475468 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 13:59:01.477408 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 13:59:01.542593 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 13:59:01.591230 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 13:59:01.602756 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 13:59:01.663949 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 13:59:01.664837 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:01.801244 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 13:59:01.807861 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 13:59:01.812851 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:01.816022 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:01.816876 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:01.820647 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:59:01.824679 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 13:59:01.833374 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:01.836189 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:01.839162 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:01.913990 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 13:59:01.935581 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 13:59:01.992564 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 13:59:02.050783 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 13:59:02.051592 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 13:59:02.062637 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 13:59:02.110821 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 13:59:02.216217 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:59:02.232744 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:59:02.237283 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 13:59:02.238449 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:02.246349 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:02.250257 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:59:02.252991 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 13:59:02.260612 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 13:59:02.278093 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:02.285390 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:02.288209 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:02.293439 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 13:59:02.317206 15426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 116 similar messages]
W20240503 13:59:02.331987 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 13:59:02.345516 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 13:59:02.461930 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 13:59:02.466975 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 13:59:02.479563 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 13:59:02.488802 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 13:59:02.517819 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 13:59:02.544106 13118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240503 13:59:02.616662 15691 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 112 similar messages]
W20240503 13:59:02.618592 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:02.619179 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 13:59:02.664299 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:02.670397 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:02.677798 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:02.678032 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:02.683508 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:59:02.694180 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 13:59:02.707527 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:59:02.707629 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:59:02.719862 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 13:59:02.726590 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 13:59:02.744765 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 13:59:02.799384 13249 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 110 similar messages]
W20240503 13:59:02.818306 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 13:59:02.858553 12987 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240503 13:59:02.902679 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 13:59:02.926363 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 13:59:03.077713 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:03.083150 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:59:03.083781 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 68 similar messages]
I20240503 13:59:03.089464 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:03.094833 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:03.105531 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:03.106107 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:03.106444 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:03.121133 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 13:59:03.130201 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:03.131825 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 13:59:03.141479 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 13:59:03.182343 15558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240503 13:59:03.203986 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 13:59:03.231762 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 13:59:03.299396 12896 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240503 13:59:03.300208 12896 catalog_manager.cc:4573] TS e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2427783ee0e348b98453818704ece5f8: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:03.302250 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 13:59:03.342911 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 13:59:03.348896 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 13:59:03.381088 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 13:59:03.408144 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 13:59:03.425500 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 13:59:03.489678 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 13:59:03.494709 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:03.494880 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:03.496158 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:03.501312 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:03.501441 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:03.505446 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:03.515758 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:03.554692 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 13:59:03.600140 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 13:59:03.615831 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 13:59:03.620028 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 13:59:03.628587 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 13:59:03.630429 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 13:59:03.642969 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 13:59:03.732543 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 13:59:03.738623 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:03.740902 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 13:59:03.741842 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 13:59:03.750473 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 13:59:03.797340 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 13:59:03.871327 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 13:59:03.917600 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 13:59:03.923942 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 13:59:03.927264 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:03.932432 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:03.932662 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:03.934042 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:03.938791 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:03.939021 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:03.962021 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:03.967968 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 13:59:04.053642 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 13:59:04.060168 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 13:59:04.139339 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 13:59:04.169838 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 13:59:04.255003 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 13:59:04.276952 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 13:59:04.286276 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 13:59:04.327688 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:04.343768 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 13:59:04.350507 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:04.356390 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:04.360242 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:04.365762 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:04.366920 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:04.369181 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:04.383348 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 13:59:04.389209 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:59:04.474071 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 13:59:04.521499 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 13:59:04.529906 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 13:59:04.534118 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 13:59:04.537242 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 13:59:04.541276 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 13:59:04.566537 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 13:59:04.648356 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 13:59:04.686136 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 13:59:04.719236 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 13:59:04.757527 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:04.761862 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:04.764412 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:59:04.767119 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 13:59:04.772470 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:04.776122 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:04.779503 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 13:59:04.793308 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 13:59:04.813108 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:04.844931 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:04.900703 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 13:59:04.971231 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 13:59:04.993755 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 13:59:05.084439 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 13:59:05.184335 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 13:59:05.188841 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:05.190361 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:05.192613 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:05.193619 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:05.199187 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:59:05.213608 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 13:59:05.215006 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:05.220933 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:05.241421 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 13:59:05.254354 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 13:59:05.265688 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 13:59:05.297441 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:05.328857 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 13:59:05.331116 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 13:59:05.426836 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 13:59:05.526083 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 13:59:05.533450 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 13:59:05.561069 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 13:59:05.601295 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 13:59:05.631181 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:05.634681 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:05.642319 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:05.644306 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:59:05.646660 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 13:59:05.650599 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:59:05.655287 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:59:05.659906 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 13:59:05.660590 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:59:05.688774 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:05.708217 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 13:59:05.725718 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 13:59:05.815366 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 13:59:05.838126 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 13:59:05.870790 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 13:59:05.887748 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 13:59:05.932548 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 13:59:05.933096 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 13:59:05.988440 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 13:59:06.011704 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 13:59:06.048157 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:06.049943 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:06.052423 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:59:06.055116 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 13:59:06.063659 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:06.064882 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:06.065444 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:06.081450 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:06.088922 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 13:59:06.182334 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 13:59:06.191016 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 13:59:06.193130 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 13:59:06.207048 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:06.237699 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 13:59:06.301218 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 13:59:06.419255 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 13:59:06.422485 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 13:59:06.425217 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 13:59:06.458724 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:06.462323 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:59:06.466735 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 13:59:06.470356 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:06.475469 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:06.476631 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:06.478207 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:06.502038 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:06.525381 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 13:59:06.616137 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:06.621858 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 13:59:06.662976 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 13:59:06.705897 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 13:59:06.716240 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 13:59:06.717645 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 13:59:06.761806 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 13:59:06.816058 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 13:59:06.816742 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 13:59:06.816771 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 13:59:06.872025 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:06.876454 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:06.885191 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:06.892649 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:06.901152 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:06.901733 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:59:06.908522 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:59:06.932262 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 13:59:06.941485 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 13:59:06.955857 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 13:59:06.967332 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 13:59:07.037480 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 13:59:07.109517 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 13:59:07.111752 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 13:59:07.157588 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 13:59:07.161453 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 13:59:07.210660 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 13:59:07.216610 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:07.232367 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 13:59:07.280023 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 13:59:07.285022 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 13:59:07.305757 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:07.306573 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:07.311017 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:07.325655 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:59:07.332078 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:07.337778 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:59:07.368618 15426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240503 13:59:07.369351 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 13:59:07.372952 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:59:07.421618 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 13:59:07.430881 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 13:59:07.478513 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 13:59:07.561448 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 13:59:07.562146 13118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 57 similar messages]
W20240503 13:59:07.597151 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:07.600593 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 13:59:07.632990 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 13:59:07.641571 15691 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 108 similar messages]
W20240503 13:59:07.681124 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 13:59:07.725795 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 13:59:07.729353 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 13:59:07.731352 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:59:07.736100 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 13:59:07.738837 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:07.739490 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:07.745026 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:59:07.747017 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:07.780766 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:59:07.803697 13249 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 108 similar messages]
W20240503 13:59:07.872467 12987 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240503 13:59:07.982937 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 13:59:08.021245 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 13:59:08.044430 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 13:59:08.077483 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 13:59:08.103153 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240503 13:59:08.103914 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 13:59:08.116369 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 13:59:08.123754 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 13:59:08.146407 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:08.150635 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:08.153414 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:08.156687 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:08.157882 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:08.158912 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:08.169816 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:59:08.204193 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:59:08.205566 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 13:59:08.212713 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 13:59:08.213002 15558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240503 13:59:08.213629 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 13:59:08.222668 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 13:59:08.296990 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 13:59:08.373261 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 13:59:08.400833 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 13:59:08.454106 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 13:59:08.487928 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 13:59:08.509300 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 13:59:08.546437 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 13:59:08.574761 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:59:08.578070 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 13:59:08.583319 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 13:59:08.585145 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:08.585296 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:08.585542 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:08.586405 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:59:08.598080 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 13:59:08.602847 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 13:59:08.611636 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:08.623531 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:08.694062 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 13:59:08.736140 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 13:59:08.770521 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 13:59:08.802389 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 13:59:08.854228 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 13:59:08.894626 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 13:59:08.985291 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:08.988175 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:08.994693 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:59:08.998757 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 13:59:09.004074 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:09.005851 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:09.012429 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:09.015522 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:09.051553 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 13:59:09.112361 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 13:59:09.125680 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 13:59:09.149657 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 13:59:09.149992 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:09.225255 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 13:59:09.226562 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 13:59:09.234752 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 13:59:09.263023 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 13:59:09.272447 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 13:59:09.305727 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 13:59:09.324164 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 13:59:09.368405 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 13:59:09.398074 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 13:59:09.399703 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:59:09.402858 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 13:59:09.403923 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:09.405194 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:09.406251 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:09.426265 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:59:09.432711 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:59:09.437752 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 13:59:09.442166 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 13:59:09.459117 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:59:09.567842 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 13:59:09.591895 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 13:59:09.680284 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 13:59:09.681300 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 13:59:09.719213 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 13:59:09.770730 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:09.787782 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 13:59:09.815405 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 13:59:09.823570 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:09.829681 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:09.833714 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:09.836694 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:09.840708 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:09.841971 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:09.843065 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:09.869499 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 13:59:09.911916 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 13:59:09.940516 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 13:59:09.976917 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 13:59:10.051441 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 13:59:10.063431 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 13:59:10.098721 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 13:59:10.133291 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 13:59:10.141296 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 13:59:10.205459 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 13:59:10.255651 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:10.258301 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:10.261162 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:10.267916 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:10.285254 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:10.285394 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:59:10.292763 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:59:10.299253 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:10.329054 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 13:59:10.332469 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 13:59:10.367854 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 13:59:10.405520 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 13:59:10.498498 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 13:59:10.529698 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 13:59:10.639736 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 13:59:10.648658 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 13:59:10.675946 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:10.677095 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:10.678954 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:10.679404 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:10.682101 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:10.684106 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:59:10.701609 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 13:59:10.718003 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:10.720597 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 13:59:10.754767 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 13:59:10.759635 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:10.858500 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 13:59:10.913311 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 13:59:10.917817 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 13:59:10.965165 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 13:59:10.981422 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 13:59:10.986927 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 13:59:11.018556 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 13:59:11.041249 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 13:59:11.078019 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:11.082032 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:11.083716 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:59:11.090245 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 13:59:11.093151 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:11.093438 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:11.103788 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:11.109452 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:11.238550 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:11.259516 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 13:59:11.328182 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 13:59:11.361352 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 13:59:11.390610 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 13:59:11.400887 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 13:59:11.414350 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 13:59:11.434312 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 13:59:11.441246 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 13:59:11.443753 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 13:59:11.485734 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 13:59:11.499159 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:11.503270 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:11.506345 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:59:11.509713 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 13:59:11.510974 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:11.515594 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:11.526437 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:11.541040 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:11.591282 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 13:59:11.618655 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 13:59:11.628374 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 13:59:11.633038 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 13:59:11.634156 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 13:59:11.634706 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 13:59:11.720346 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 13:59:11.752849 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 13:59:11.783212 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:11.825578 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 13:59:11.830441 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 13:59:11.839583 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 13:59:11.904887 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:11.921406 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:11.922848 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:11.923617 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:11.935179 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:11.945034 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:59:11.945737 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:59:12.034132 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 13:59:12.060144 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 13:59:12.079107 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 13:59:12.086236 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 13:59:12.196292 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:12.237250 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 13:59:12.260056 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 13:59:12.285997 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 13:59:12.302013 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 13:59:12.323727 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 13:59:12.336741 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:12.344327 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:12.346206 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:59:12.348682 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:12.350821 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:59:12.369112 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 13:59:12.370079 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:59:12.372731 15426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 121 similar messages]
I20240503 13:59:12.395413 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:59:12.406297 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 13:59:12.432538 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 13:59:12.482066 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 13:59:12.494482 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 13:59:12.496207 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 13:59:12.542388 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 13:59:12.558501 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 13:59:12.572317 13118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240503 13:59:12.602501 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 13:59:12.646361 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:12.654964 15691 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 107 similar messages]
W20240503 13:59:12.657003 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 13:59:12.691447 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 13:59:12.751821 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:12.756857 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:12.756933 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:12.760147 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:12.762002 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:12.768275 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:12.804664 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:12.806874 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 13:59:12.882634 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 13:59:12.885684 12987 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240503 13:59:12.886246 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 13:59:12.887286 13249 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 110 similar messages]
W20240503 13:59:12.960788 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 13:59:13.000931 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 13:59:13.027770 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:13.037361 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 13:59:13.071121 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 13:59:13.079588 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 13:59:13.084229 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 13:59:13.159797 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 72 similar messages]
I20240503 13:59:13.162372 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:13.170401 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:13.179867 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 13:59:13.180670 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 13:59:13.183755 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:59:13.188771 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:13.189317 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:13.194496 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:59:13.228899 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 13:59:13.250154 15558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240503 13:59:13.279289 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 13:59:13.427383 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:13.428088 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 13:59:13.486197 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 13:59:13.504681 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 13:59:13.548722 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 13:59:13.568409 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 13:59:13.585853 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:13.590718 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:59:13.591439 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 13:59:13.598348 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:59:13.601254 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 13:59:13.607968 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:13.608474 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:59:13.611305 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:13.629068 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:59:13.638572 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 13:59:13.677336 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 13:59:13.736912 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 13:59:13.793264 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 13:59:13.889293 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 13:59:13.919308 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 13:59:13.932092 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 13:59:13.964973 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 13:59:13.991664 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 13:59:13.996302 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:14.000200 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:14.000236 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:14.005704 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:59:14.010882 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 13:59:14.017414 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:14.019157 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:59:14.042416 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:59:14.044061 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 13:59:14.046022 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:14.115635 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 13:59:14.140352 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 13:59:14.174682 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 13:59:14.181679 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 13:59:14.182200 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 13:59:14.255259 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 13:59:14.270579 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 13:59:14.324339 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 13:59:14.342988 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 13:59:14.398566 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:59:14.400379 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 13:59:14.406994 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:14.408627 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:59:14.410944 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 13:59:14.417969 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:14.421288 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:14.423216 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 13:59:14.427054 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:14.456473 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:59:14.541848 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 13:59:14.594185 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:14.618613 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 13:59:14.641309 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 13:59:14.653333 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 13:59:14.800189 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 13:59:14.814750 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:59:14.817221 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 13:59:14.819430 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 13:59:14.819592 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:14.828611 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:14.829011 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:14.834594 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:59:14.837697 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 13:59:14.842753 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:59:14.843132 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:59:14.846957 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 13:59:14.858140 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 13:59:14.927354 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 13:59:14.942467 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 13:59:15.062323 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 13:59:15.062752 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 13:59:15.064790 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 13:59:15.082942 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 13:59:15.107749 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 13:59:15.125926 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 13:59:15.146549 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 13:59:15.178215 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:15.195612 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 13:59:15.235591 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 13:59:15.237093 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 13:59:15.240710 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:15.249262 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:15.257674 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:15.257877 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:15.262430 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:15.281569 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:15.334791 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 13:59:15.363243 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 13:59:15.427193 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 13:59:15.463908 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 13:59:15.509423 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 13:59:15.547070 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 13:59:15.609194 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 13:59:15.628537 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:15.629283 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 13:59:15.634107 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 13:59:15.659276 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:15.669426 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:15.671705 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:15.680035 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:15.680852 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:59:15.689749 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 13:59:15.695725 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:59:15.701133 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:59:15.797108 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 13:59:15.809635 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 13:59:15.812958 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 13:59:15.949978 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 13:59:16.034684 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 13:59:16.056589 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:16.063584 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:16.066315 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:16.068259 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:16.072557 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:59:16.072613 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:16.095842 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:59:16.100232 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 13:59:16.112277 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:16.142314 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 13:59:16.142397 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 13:59:16.202435 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 13:59:16.271134 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 13:59:16.287875 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 13:59:16.308137 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 13:59:16.331566 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 13:59:16.385566 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 13:59:16.388571 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 13:59:16.444440 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 13:59:16.459770 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 13:59:16.465258 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 13:59:16.466369 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 13:59:16.469452 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:16.471562 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:16.471866 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:16.472807 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:16.474254 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:16.480062 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:16.485226 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 13:59:16.510277 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:59:16.512677 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 13:59:16.618534 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 13:59:16.618614 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 13:59:16.657200 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 13:59:16.665468 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 13:59:16.679632 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 13:59:16.687608 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:16.689895 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 13:59:16.736919 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 13:59:16.862601 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:16.868288 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:16.871661 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:16.880753 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:16.884955 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:16.905694 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:16.914460 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:16.917093 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 13:59:16.983619 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 13:59:17.052814 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 13:59:17.092754 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 13:59:17.143527 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 13:59:17.161397 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 13:59:17.251288 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 13:59:17.255765 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 13:59:17.287204 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:17.290041 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:17.290769 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:17.294461 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:17.294696 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:59:17.311228 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:17.313597 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 13:59:17.315239 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:17.330701 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:17.426057 15426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240503 13:59:17.445370 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 13:59:17.456542 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 13:59:17.497304 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 13:59:17.522470 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 13:59:17.523725 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 13:59:17.544303 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 13:59:17.547861 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 13:59:17.588361 13118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240503 13:59:17.595566 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 13:59:17.598646 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 13:59:17.631644 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 13:59:17.662032 15691 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 107 similar messages]
W20240503 13:59:17.680887 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 13:59:17.701473 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 13:59:17.704021 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:17.704636 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:17.709148 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:17.717312 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:17.717962 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:17.720597 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:17.734997 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:17.805997 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 13:59:17.809758 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 13:59:17.895814 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 13:59:17.925375 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:17.933852 12987 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240503 13:59:17.940263 13249 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 110 similar messages]
W20240503 13:59:17.956256 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 13:59:17.982645 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 13:59:18.001837 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 13:59:18.045001 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 13:59:18.071943 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 13:59:18.105836 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 13:59:18.106523 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:18.107985 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:18.110414 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:18.117027 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:18.124660 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:18.131238 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:59:18.142024 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 13:59:18.160104 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:18.163205 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 13:59:18.165495 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 13:59:18.186890 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240503 13:59:18.187520 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 13:59:18.202963 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 13:59:18.256264 15558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240503 13:59:18.266384 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 13:59:18.401145 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 13:59:18.457707 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 13:59:18.482429 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:59:18.505877 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:18.509363 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:18.511917 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:18.511956 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:59:18.513970 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 13:59:18.528882 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 13:59:18.529876 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:18.534760 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:59:18.558272 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 13:59:18.566968 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:18.580652 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 13:59:18.630173 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 13:59:18.672266 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 13:59:18.743695 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 13:59:18.820344 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 13:59:18.878216 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 13:59:18.910874 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 13:59:18.929886 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 13:59:18.931631 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:18.932655 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 13:59:18.944861 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:59:18.945658 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:18.947429 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:18.949018 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:59:18.949960 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:18.977324 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:59:19.026710 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 13:59:19.027711 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 13:59:19.077548 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 13:59:19.088745 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 13:59:19.099856 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 13:59:19.160423 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 13:59:19.181020 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 13:59:19.182934 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 13:59:19.186810 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 13:59:19.235219 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 13:59:19.310600 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 13:59:19.324414 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:59:19.339671 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:19.340698 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:19.348623 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:19.352659 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:59:19.357496 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 13:59:19.358106 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 13:59:19.360972 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:59:19.368749 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 13:59:19.396960 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:19.401408 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:19.403017 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 13:59:19.454624 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 13:59:19.461767 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 13:59:19.554694 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 13:59:19.641511 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 13:59:19.718688 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 13:59:19.750870 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:19.753396 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:19.759891 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:19.760211 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:59:19.765338 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:59:19.770330 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:19.789428 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:59:19.789577 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:59:19.812858 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 13:59:19.837384 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 13:59:19.848090 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 13:59:19.904177 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 13:59:19.910627 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 13:59:19.942394 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 13:59:19.972640 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 13:59:20.037249 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 13:59:20.041420 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 13:59:20.047994 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 13:59:20.072329 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 13:59:20.113548 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 13:59:20.123798 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 13:59:20.151351 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 13:59:20.172690 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:20.180881 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:20.181608 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:20.182026 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:59:20.190402 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 13:59:20.192397 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:20.201400 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:20.211045 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:20.220449 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 13:59:20.277945 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:20.302728 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 13:59:20.334750 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 13:59:20.351039 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 13:59:20.392129 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 13:59:20.532003 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 13:59:20.553663 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 13:59:20.560403 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 13:59:20.569550 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 13:59:20.570421 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 13:59:20.579743 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:20.584038 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:59:20.587934 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 13:59:20.589848 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:20.593531 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:20.604151 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:20.611035 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:20.626796 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:20.658020 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 13:59:20.708093 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:20.727043 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 13:59:20.843566 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 13:59:20.858628 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 13:59:20.921453 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 13:59:20.974663 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 13:59:20.997447 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:20.997730 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:20.997999 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:21.003228 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:21.005738 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:21.008991 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:21.029685 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:21.046986 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 13:59:21.088337 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 13:59:21.096453 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:21.098819 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 13:59:21.157730 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 13:59:21.287919 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 13:59:21.335711 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 13:59:21.336020 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 13:59:21.366421 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 13:59:21.399690 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:21.404682 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:59:21.407518 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 13:59:21.409395 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 13:59:21.410455 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:21.415216 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:21.415638 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:59:21.444283 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:21.444593 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:59:21.468724 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 13:59:21.476995 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 13:59:21.536715 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 13:59:21.548990 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 13:59:21.621021 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 13:59:21.717653 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:21.741305 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 13:59:21.768865 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 13:59:21.777035 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 13:59:21.814308 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:21.821547 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:21.823447 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:21.824563 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:21.826650 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:21.829016 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:21.834067 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:21.878221 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 13:59:21.893067 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 13:59:21.893957 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 13:59:21.900733 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 13:59:21.961505 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 13:59:21.966506 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 13:59:22.042259 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 13:59:22.049130 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 13:59:22.061295 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 13:59:22.129258 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:22.151124 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 13:59:22.184244 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 13:59:22.198777 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 13:59:22.221928 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:22.232329 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:22.232856 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:22.246431 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:22.248288 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:59:22.253466 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:59:22.253805 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 13:59:22.267515 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:59:22.293339 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 13:59:22.321060 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 13:59:22.326159 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 13:59:22.451771 15426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240503 13:59:22.488186 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 13:59:22.504369 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 13:59:22.567193 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 13:59:22.587810 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 13:59:22.600230 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 13:59:22.605723 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 13:59:22.653523 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 13:59:22.653604 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:59:22.654498 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:22.659591 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:22.665251 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:22.668641 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:22.670743 15691 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 110 similar messages]
W20240503 13:59:22.676589 13118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240503 13:59:22.676990 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 13:59:22.677454 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 13:59:22.679499 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:22.685472 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:59:22.725697 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 13:59:22.730873 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 13:59:22.757524 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 13:59:22.917634 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 13:59:22.985420 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 13:59:22.991312 13249 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 109 similar messages]
W20240503 13:59:23.010038 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 13:59:23.014223 12987 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240503 13:59:23.027433 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 13:59:23.040817 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 13:59:23.050987 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:23.056219 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 13:59:23.068257 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 13:59:23.080682 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:23.083706 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 13:59:23.084220 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 13:59:23.087288 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:23.093413 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:23.096927 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:23.099812 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:23.108891 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:23.110361 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 13:59:23.127889 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 13:59:23.153237 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 13:59:23.182655 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 13:59:23.198841 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240503 13:59:23.199445 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 13:59:23.225632 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 13:59:23.333797 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 13:59:23.337307 15558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240503 13:59:23.425246 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 13:59:23.446003 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:23.474591 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 13:59:23.493263 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 13:59:23.495219 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:23.505040 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:23.510360 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:23.511147 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:23.511157 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:23.513968 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:59:23.530369 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 13:59:23.535775 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:23.545161 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 13:59:23.545253 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 13:59:23.630934 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 13:59:23.764307 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 13:59:23.872334 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 13:59:23.899225 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:23.900202 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:59:23.904145 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 13:59:23.914166 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:23.914475 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:23.917713 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:23.925474 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:59:23.926079 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:59:23.940662 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 13:59:23.951454 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 13:59:23.954535 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 13:59:23.962046 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 13:59:23.982344 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:24.022809 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 13:59:24.140321 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 13:59:24.171643 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 13:59:24.188450 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 13:59:24.338423 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:24.338429 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:24.340591 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:24.343284 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:24.356642 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:24.361232 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:24.374799 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 13:59:24.396231 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:59:24.398047 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 13:59:24.398520 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 13:59:24.405107 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 13:59:24.435766 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 13:59:24.436878 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 13:59:24.465713 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 13:59:24.481664 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 13:59:24.489475 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 13:59:24.523641 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 13:59:24.527547 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:24.552688 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 13:59:24.569702 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 13:59:24.706305 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 13:59:24.740873 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:24.743505 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:24.747797 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:24.754297 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:24.758510 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:59:24.762499 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:24.783385 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:59:24.793444 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 13:59:24.800071 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 13:59:24.837795 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 13:59:24.865214 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 13:59:24.950623 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 13:59:24.991518 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 13:59:24.997786 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:25.002593 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 13:59:25.002808 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 13:59:25.110651 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 13:59:25.146212 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:25.148936 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 13:59:25.152554 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 13:59:25.152773 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:25.157272 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:25.166232 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:25.166559 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:59:25.171417 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:59:25.189865 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 13:59:25.229535 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 13:59:25.237926 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 13:59:25.262881 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 13:59:25.352041 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 13:59:25.361299 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 13:59:25.385558 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 13:59:25.394328 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:25.399030 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 13:59:25.405336 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 13:59:25.435258 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 13:59:25.542892 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 13:59:25.560081 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:25.560271 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:59:25.560950 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 13:59:25.573877 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:59:25.577787 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 13:59:25.578333 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:25.584667 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:59:25.594357 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 13:59:25.596688 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:59:25.598812 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:59:25.643383 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 13:59:25.693972 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 13:59:25.703346 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 13:59:25.725739 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 13:59:25.781435 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 13:59:25.781827 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 13:59:25.793761 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 13:59:25.817787 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 13:59:25.820008 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 13:59:25.871376 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:25.940639 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 13:59:25.950506 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 13:59:25.978252 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:25.981557 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:25.991758 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:25.994203 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:25.996039 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:26.011078 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:59:26.014256 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:59:26.030304 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 13:59:26.133433 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 13:59:26.229285 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 13:59:26.268796 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 13:59:26.276475 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 13:59:26.291744 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 13:59:26.310178 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 13:59:26.392102 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:26.397094 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:26.398337 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:26.414286 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:26.416862 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:26.426687 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 13:59:26.428200 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:59:26.439888 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 13:59:26.439992 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 13:59:26.442813 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:59:26.489933 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:26.615792 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 13:59:26.684252 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 13:59:26.739140 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 13:59:26.784582 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 13:59:26.793978 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 13:59:26.809090 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:26.813890 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:26.814136 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:26.815366 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:26.817767 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:26.831956 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:59:26.842862 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:59:26.851615 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 13:59:26.892987 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 13:59:26.920042 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 13:59:26.927644 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 13:59:26.950724 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 13:59:26.957749 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:27.031509 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 13:59:27.048249 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 13:59:27.081530 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 13:59:27.089898 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 13:59:27.126871 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 13:59:27.223958 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:27.225194 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:27.226377 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:27.226815 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:27.229686 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 13:59:27.237969 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 13:59:27.246511 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:59:27.268959 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:59:27.321559 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 13:59:27.332320 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 13:59:27.351747 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:27.356773 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 13:59:27.412818 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 13:59:27.478274 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 13:59:27.482314 15426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 118 similar messages]
W20240503 13:59:27.504566 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 13:59:27.544363 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 13:59:27.550868 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 13:59:27.571508 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 13:59:27.623422 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 13:59:27.641600 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:59:27.642545 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 13:59:27.643785 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:59:27.646644 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 13:59:27.649122 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:27.653818 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:59:27.653941 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:27.656841 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:27.662302 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:59:27.676717 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 13:59:27.681263 15691 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 107 similar messages]
W20240503 13:59:27.682054 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 13:59:27.735916 13118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240503 13:59:27.864642 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 13:59:27.893695 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 13:59:27.929926 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 13:59:27.946362 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:27.998327 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 13:59:28.029938 12987 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 70 similar messages]
I20240503 13:59:28.048251 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:28.049059 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:28.051182 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:59:28.051357 13249 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 111 similar messages]
W20240503 13:59:28.056350 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 13:59:28.057492 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 13:59:28.059896 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:28.060125 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:59:28.062512 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:59:28.064697 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 13:59:28.072979 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 13:59:28.073544 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:59:28.085198 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 13:59:28.092607 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 13:59:28.121124 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 13:59:28.178557 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 13:59:28.229441 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 13:59:28.297199 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 13:59:28.299651 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 13:59:28.318089 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 13:59:28.331773 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 13:59:28.333444 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 13:59:28.339219 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240503 13:59:28.340179 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 13:59:28.344029 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 13:59:28.347949 15558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240503 13:59:28.352525 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:28.397486 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 13:59:28.445104 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:28.450975 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:28.451905 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:28.456840 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:28.457195 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:28.469898 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:28.481173 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:28.510991 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 13:59:28.632215 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 13:59:28.654697 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 13:59:28.683426 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 13:59:28.684829 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 13:59:28.781031 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 13:59:28.805411 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 13:59:28.861940 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:28.862628 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:28.864656 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:28.873337 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:28.885036 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:28.887789 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:28.919906 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:28.929454 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 13:59:28.972887 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 13:59:28.975585 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:29.073225 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 13:59:29.115978 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 13:59:29.122627 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 13:59:29.132938 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 13:59:29.141693 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 13:59:29.269953 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 13:59:29.302408 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:29.309427 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:29.313527 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:29.318686 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:29.322650 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:29.325683 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:59:29.349100 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 13:59:29.356072 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:29.376417 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 13:59:29.399746 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 13:59:29.409543 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:29.484972 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 13:59:29.493656 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 13:59:29.505807 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 13:59:29.587306 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 13:59:29.599516 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 13:59:29.615521 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 13:59:29.666879 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 13:59:29.718230 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:29.719398 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:29.726653 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:29.727703 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:29.738857 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:29.742777 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:59:29.746414 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:59:29.774317 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 13:59:29.792397 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 13:59:29.801074 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 13:59:29.802698 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 13:59:29.811625 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 13:59:29.819821 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 13:59:29.837353 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 13:59:29.847388 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:29.926420 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 13:59:30.026072 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 13:59:30.045238 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 13:59:30.068115 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 13:59:30.089010 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 13:59:30.116550 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 13:59:30.123247 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:30.128628 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:30.132659 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:30.133960 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:30.148727 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:30.156944 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:30.161792 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:30.203063 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 13:59:30.233039 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 13:59:30.255652 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 13:59:30.277976 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 13:59:30.299561 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:30.317044 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 13:59:30.317303 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 13:59:30.339586 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 13:59:30.414772 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 13:59:30.462579 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 13:59:30.479471 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 13:59:30.496217 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 13:59:30.537884 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:30.538914 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:30.543241 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:30.544895 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:30.544946 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:30.554853 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:59:30.557760 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:59:30.675472 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:30.725108 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 13:59:30.726907 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 13:59:30.735471 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 13:59:30.767748 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 13:59:30.784535 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 13:59:30.785641 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 13:59:30.810171 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 13:59:30.848254 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 13:59:30.878291 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 13:59:30.885289 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 13:59:30.895179 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 13:59:30.948343 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:30.950078 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:30.952402 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:30.961226 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:30.961839 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:59:30.963886 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:30.969833 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:59:31.018843 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 13:59:31.021134 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 13:59:31.043653 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 13:59:31.079033 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 13:59:31.116627 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 13:59:31.140436 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 13:59:31.180851 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 13:59:31.221735 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 13:59:31.260177 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:31.274825 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 13:59:31.309051 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 13:59:31.314837 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 13:59:31.324939 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 13:59:31.359486 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:31.361513 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:31.374576 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:31.378643 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:31.378913 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:31.389480 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:59:31.393635 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 13:59:31.400245 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:31.417560 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 13:59:31.586201 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 13:59:31.670562 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 13:59:31.672502 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:31.691071 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 13:59:31.751425 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 13:59:31.757258 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:31.762315 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:31.762754 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:59:31.773592 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 13:59:31.777474 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:31.779217 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:31.792645 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:59:31.797577 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:59:31.821853 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 13:59:31.956737 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 13:59:31.977442 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 13:59:32.015949 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 13:59:32.027302 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 13:59:32.045763 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 13:59:32.060180 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 13:59:32.150933 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 13:59:32.158008 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:59:32.172868 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:32.177217 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:32.180320 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:32.187748 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:32.190147 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:32.198951 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 13:59:32.207587 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:59:32.218761 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 13:59:32.225337 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 13:59:32.225778 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:59:32.299006 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 13:59:32.299772 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 13:59:32.348752 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 13:59:32.357393 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 13:59:32.429601 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 13:59:32.465766 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 13:59:32.491393 15426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240503 13:59:32.540350 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 13:59:32.553438 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 13:59:32.583863 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:32.583863 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:32.588529 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:32.596547 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:32.599054 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:32.606770 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:32.618232 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:32.646983 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 13:59:32.647330 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 13:59:32.662222 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 13:59:32.693960 15691 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 107 similar messages]
W20240503 13:59:32.702172 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 13:59:32.710557 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 13:59:32.739132 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:32.814785 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 13:59:32.814901 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 13:59:32.825438 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 13:59:32.897862 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 13:59:32.912066 13118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240503 13:59:32.912928 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 13:59:32.956066 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 13:59:32.968952 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 13:59:33.003672 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 13:59:33.017048 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 13:59:33.020692 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 13:59:33.024448 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:33.025341 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:33.029623 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:59:33.030092 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:33.042744 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:33.047417 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:59:33.055593 12987 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 67 similar messages]
W20240503 13:59:33.132419 13249 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 113 similar messages]
W20240503 13:59:33.175117 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 13:59:33.205511 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 13:59:33.233259 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:33.246234 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 13:59:33.274099 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 13:59:33.284057 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 13:59:33.303673 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 13:59:33.357331 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 13:59:33.409024 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 13:59:33.412791 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 13:59:33.417865 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:33.421902 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:33.425566 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:33.425802 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:33.426697 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:59:33.430716 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:33.437268 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:59:33.447944 15558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240503 13:59:33.460927 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 13:59:33.462325 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 13:59:33.505702 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240503 13:59:33.506841 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 13:59:33.611147 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 13:59:33.640719 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:33.660135 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 13:59:33.672842 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 13:59:33.691321 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 13:59:33.696900 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 13:59:33.701157 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 13:59:33.703955 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 13:59:33.738756 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 13:59:33.746434 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 13:59:33.801959 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 13:59:33.824816 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:33.826658 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:33.835403 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:33.837514 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:33.838280 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:59:33.846285 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 13:59:33.846788 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:59:33.847909 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:59:34.046087 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 13:59:34.046681 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 13:59:34.052112 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:34.101871 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 13:59:34.113210 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 13:59:34.122395 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 13:59:34.151283 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 13:59:34.227136 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:34.229092 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:34.240020 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:34.253015 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:34.259922 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:34.268733 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:34.296094 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:34.349081 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 13:59:34.363633 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 13:59:34.530370 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 13:59:34.607685 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:34.641989 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 13:59:34.661203 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 13:59:34.665130 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:34.665944 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:34.666132 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:34.670120 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:34.674729 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:34.695847 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:59:34.705365 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 13:59:34.715201 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 13:59:34.715991 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 13:59:34.716578 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:34.721267 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 13:59:34.753119 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 13:59:34.765031 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 13:59:34.809659 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 13:59:34.824117 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 13:59:34.874153 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 13:59:34.900377 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 13:59:35.058372 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 13:59:35.076052 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:35.079021 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:35.082301 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:35.084638 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:59:35.085714 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 13:59:35.088982 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:35.117947 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:35.120911 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:35.132440 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 13:59:35.139827 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 13:59:35.140023 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 13:59:35.186189 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:35.200553 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 13:59:35.220892 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 13:59:35.277197 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 13:59:35.295657 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 13:59:35.306113 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 13:59:35.313642 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 13:59:35.338377 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 13:59:35.379941 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 13:59:35.410257 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 13:59:35.440909 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 13:59:35.441329 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 13:59:35.491191 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:35.491725 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:35.497810 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:35.499727 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:35.500635 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:59:35.509891 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:35.511695 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:59:35.621459 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 13:59:35.625507 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 13:59:35.647800 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 13:59:35.672927 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 13:59:35.751960 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 13:59:35.785410 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 13:59:35.796094 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:35.802716 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 13:59:35.803551 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 13:59:35.821566 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 13:59:35.846482 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 13:59:35.893062 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:35.896835 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:35.899015 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:59:35.901777 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 13:59:35.903421 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:59:35.910481 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 13:59:35.913889 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 13:59:35.915488 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:59:35.936261 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:35.940052 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:59:35.940527 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 13:59:36.090909 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 13:59:36.113629 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 13:59:36.133662 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 13:59:36.140343 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 13:59:36.160085 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 13:59:36.278618 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:36.289765 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 13:59:36.292865 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 13:59:36.329995 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:36.340811 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:36.344617 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:59:36.346984 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:36.349495 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:36.351565 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:59:36.353116 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 13:59:36.364564 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 13:59:36.370580 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:59:36.379024 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 13:59:36.460357 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 13:59:36.463907 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 13:59:36.484645 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 13:59:36.506603 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 13:59:36.559844 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 13:59:36.626271 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 13:59:36.636178 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 13:59:36.741428 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:36.741667 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:36.747125 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:36.748548 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:36.749423 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:59:36.750696 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:36.751761 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:36.785758 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:59:36.842929 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 13:59:36.959278 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 13:59:37.057523 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 13:59:37.069376 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 13:59:37.133620 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 13:59:37.148209 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:37.151115 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:59:37.158810 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 13:59:37.159582 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:37.164778 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:37.166765 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:59:37.168638 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:37.170707 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:59:37.173663 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 13:59:37.218505 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 13:59:37.232666 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 13:59:37.246925 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:37.260353 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 13:59:37.260813 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 13:59:37.310688 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 13:59:37.334998 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 13:59:37.461766 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 13:59:37.477775 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 13:59:37.527105 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 13:59:37.531929 15426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 122 similar messages]
W20240503 13:59:37.532084 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 13:59:37.548031 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 13:59:37.565569 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:37.569765 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:37.571669 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:37.577682 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:37.579254 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:37.592715 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:37.596701 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:37.621685 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 13:59:37.647372 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 13:59:37.748981 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 13:59:37.761827 15691 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 110 similar messages]
W20240503 13:59:37.766147 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 13:59:37.781216 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 13:59:37.814198 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:37.826558 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 13:59:37.854597 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 13:59:37.857178 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 13:59:37.865795 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 13:59:37.902055 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 13:59:37.914184 13118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240503 13:59:37.927165 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 13:59:37.932560 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 13:59:37.943445 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 13:59:37.972254 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 13:59:37.988966 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 13:59:37.995955 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:38.003260 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:38.006456 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:38.008590 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:59:38.010649 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:38.012650 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:38.014619 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:59:38.068562 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 13:59:38.073094 12987 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240503 13:59:38.103817 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 13:59:38.111184 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 13:59:38.165001 13249 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 110 similar messages]
W20240503 13:59:38.207127 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 13:59:38.254241 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 13:59:38.320029 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:38.328953 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 13:59:38.346904 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 13:59:38.354157 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 13:59:38.357988 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 13:59:38.387202 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 13:59:38.397572 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 13:59:38.403812 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:38.409833 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:38.412393 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:38.415434 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:38.426391 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:38.427214 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:38.433924 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:38.459246 15558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240503 13:59:38.459877 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 13:59:38.559581 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240503 13:59:38.576269 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 13:59:38.667127 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 13:59:38.671653 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 13:59:38.713387 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 13:59:38.715024 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 13:59:38.765141 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 13:59:38.778374 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 13:59:38.786758 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 13:59:38.798348 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 13:59:38.811849 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:59:38.814265 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:59:38.818073 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:38.824641 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:38.825167 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:38.828282 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:59:38.843987 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 13:59:38.846272 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:38.854928 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:38.856931 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 13:59:38.902441 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 13:59:38.941371 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 13:59:38.941545 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 13:59:39.099203 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 13:59:39.125005 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 13:59:39.219025 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 13:59:39.222354 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:39.229521 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:39.237903 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:39.245170 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:39.246270 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:39.254575 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:39.280946 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:39.370329 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:39.380645 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 13:59:39.454250 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 13:59:39.538436 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 13:59:39.548990 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 13:59:39.568562 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 13:59:39.625164 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 13:59:39.651755 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:39.655346 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:39.664815 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:39.665482 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:59:39.666147 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:59:39.669742 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 13:59:39.673542 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 13:59:39.677577 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:39.679841 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:59:39.700284 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 13:59:39.751827 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 13:59:39.818760 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 13:59:39.847703 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 13:59:39.879469 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 13:59:39.927865 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 13:59:39.930856 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 13:59:39.977782 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:40.025117 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 13:59:40.056267 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:40.057185 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:40.066730 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:59:40.068172 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 13:59:40.072562 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:40.079807 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:40.080338 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:40.085181 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:40.167022 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 13:59:40.218897 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 13:59:40.269273 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 13:59:40.276463 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 13:59:40.319105 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 13:59:40.391908 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 13:59:40.394846 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 13:59:40.449824 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 13:59:40.460143 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:40.460798 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:40.463807 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:40.469159 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:40.470306 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:40.471139 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:40.473743 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 13:59:40.485915 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 13:59:40.486873 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 13:59:40.496791 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:59:40.499881 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:40.587211 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 13:59:40.588707 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 13:59:40.621865 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 13:59:40.648603 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 13:59:40.654474 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 13:59:40.660365 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 13:59:40.679773 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 13:59:40.684846 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 13:59:40.795485 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 13:59:40.820029 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 13:59:40.858227 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 13:59:40.860105 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 13:59:40.868587 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:40.874552 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:40.878206 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:40.882014 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:40.887501 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:40.896293 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:40.897358 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:40.902494 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 13:59:40.924357 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 13:59:40.968307 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 13:59:40.968688 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 13:59:41.014042 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:41.037562 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 13:59:41.039839 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 13:59:41.072525 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 13:59:41.091939 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 13:59:41.107631 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 13:59:41.123415 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 13:59:41.128657 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 13:59:41.140816 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 13:59:41.158493 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 13:59:41.208052 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 13:59:41.234944 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 13:59:41.254374 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 13:59:41.275417 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 13:59:41.279937 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 13:59:41.295630 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:41.297842 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:41.304395 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:41.317414 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:41.319110 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:41.323505 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:41.353168 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:41.371519 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 13:59:41.471922 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:41.594300 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 13:59:41.607069 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 13:59:41.620440 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 13:59:41.720243 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:41.723266 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:41.729312 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:41.731982 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:41.757014 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:41.767082 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:59:41.772998 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:59:41.848206 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 13:59:41.854710 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 13:59:41.888922 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 13:59:41.889278 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 13:59:41.936129 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 13:59:42.047039 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:42.057644 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 13:59:42.089116 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 13:59:42.118000 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 13:59:42.142812 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:42.143091 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:42.150700 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:59:42.153108 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 13:59:42.154631 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:42.172632 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:42.173313 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:59:42.176016 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:59:42.279094 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 13:59:42.362689 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 13:59:42.366612 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 13:59:42.402550 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 13:59:42.416656 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 13:59:42.459702 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 13:59:42.471691 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 13:59:42.479935 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 13:59:42.550946 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:42.555909 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:59:42.556083 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:59:42.557253 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:42.560681 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:42.567947 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:59:42.571593 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 13:59:42.584503 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 13:59:42.588898 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:42.588917 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:42.606165 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 13:59:42.628161 15426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 123 similar messages]
W20240503 13:59:42.642766 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 13:59:42.706279 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 13:59:42.787406 15691 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 108 similar messages]
W20240503 13:59:42.790802 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 13:59:42.828032 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 13:59:42.899483 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 13:59:42.960398 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 13:59:42.967411 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:42.970741 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:42.977231 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:42.977896 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:42.979286 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:59:42.985217 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 13:59:42.999529 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:59:43.000173 13118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 58 similar messages]
I20240503 13:59:43.014338 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:43.057539 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 13:59:43.103291 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 13:59:43.104065 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:43.107570 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 13:59:43.114276 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 13:59:43.124173 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 13:59:43.164855 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 13:59:43.169956 12987 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 73 similar messages]
W20240503 13:59:43.170516 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 13:59:43.190182 13249 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 111 similar messages]
W20240503 13:59:43.190402 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 13:59:43.190936 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 13:59:43.191139 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 13:59:43.197813 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 13:59:43.322455 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 13:59:43.358747 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 13:59:43.376333 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:43.380425 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:43.392251 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:59:43.393352 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:59:43.395340 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 13:59:43.398890 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:43.408771 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:43.416337 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:59:43.425208 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 13:59:43.426534 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 13:59:43.453603 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 13:59:43.465234 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 13:59:43.486449 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 13:59:43.524533 15558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240503 13:59:43.542637 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 13:59:43.580730 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 13:59:43.632118 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 13:59:43.640128 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 13:59:43.645232 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 13:59:43.649741 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240503 13:59:43.650311 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:43.689476 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 13:59:43.729076 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 13:59:43.783452 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:43.793350 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:43.799755 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:43.799930 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:43.805426 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:43.810336 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:59:43.823653 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 13:59:43.833241 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:43.897475 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 13:59:43.903601 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 13:59:43.909785 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 13:59:43.934651 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 13:59:43.936616 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 13:59:43.995676 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 13:59:44.066370 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:44.072458 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 13:59:44.110415 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 13:59:44.164166 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 13:59:44.192631 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:44.193961 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:44.194581 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:44.205826 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:44.206064 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:44.219517 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:44.223690 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:44.265736 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 13:59:44.298888 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 13:59:44.314325 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 13:59:44.398918 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 13:59:44.401319 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 13:59:44.524087 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 13:59:44.531122 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:59:44.606227 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:44.609344 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:44.610419 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:44.611248 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:44.611563 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:44.612557 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:44.613744 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:44.653122 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 13:59:44.676062 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 13:59:44.718999 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 13:59:44.748342 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 13:59:44.871124 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 13:59:44.947327 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 13:59:44.972985 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 13:59:44.975122 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 13:59:45.007632 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:45.009842 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:45.009913 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:59:45.010195 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 13:59:45.013581 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:59:45.017549 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 13:59:45.034946 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:59:45.036208 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:45.037995 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:59:45.102631 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 13:59:45.103825 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 13:59:45.108628 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:45.152791 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 13:59:45.293193 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 13:59:45.311288 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 13:59:45.313221 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 13:59:45.315071 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 13:59:45.379065 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 13:59:45.391999 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 13:59:45.398314 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 13:59:45.417938 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:45.418576 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:45.421123 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:45.421151 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:45.429729 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:45.431952 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:45.434727 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:45.445175 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 13:59:45.517560 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:45.533537 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 13:59:45.567327 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 13:59:45.590844 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 13:59:45.602751 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 13:59:45.651911 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 13:59:45.678130 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 13:59:45.745509 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 13:59:45.780030 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 13:59:45.780643 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 13:59:45.784932 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 13:59:45.789767 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 13:59:45.796690 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 13:59:45.838188 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 13:59:45.842288 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:45.842584 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:45.850493 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:45.858305 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:45.861793 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:45.873752 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 13:59:45.875012 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:45.875342 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:59:45.895242 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:45.913322 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 13:59:45.957029 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 13:59:46.001457 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 13:59:46.026252 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 13:59:46.048887 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 13:59:46.114961 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 13:59:46.150472 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 13:59:46.156531 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 13:59:46.191426 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 13:59:46.214738 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 13:59:46.236925 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 13:59:46.262550 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:46.265899 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:46.266244 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 13:59:46.271819 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 13:59:46.279881 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:59:46.288172 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 13:59:46.294962 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:46.295439 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:59:46.307399 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 13:59:46.318522 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:46.378820 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 13:59:46.423094 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:46.440001 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 13:59:46.459050 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 13:59:46.471463 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 13:59:46.575423 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 13:59:46.614020 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 13:59:46.673544 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 13:59:46.692194 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:46.693554 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:46.698221 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:46.700469 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:46.708711 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:59:46.714542 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:59:46.717504 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 13:59:46.736899 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 13:59:46.742203 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 13:59:46.744033 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:59:46.879810 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:46.882978 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 13:59:46.914016 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 13:59:46.956557 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 13:59:47.047571 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 13:59:47.102210 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:47.103222 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:47.105455 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:47.106879 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:59:47.109170 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 13:59:47.115003 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:59:47.131309 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:47.137492 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:59:47.150892 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 13:59:47.157377 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 13:59:47.224318 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 13:59:47.322566 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:47.434803 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 13:59:47.436717 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 13:59:47.462082 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 13:59:47.484297 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 13:59:47.513233 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:47.518013 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:47.520054 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:47.524873 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:47.537081 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:47.542451 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:47.552851 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:47.596940 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 13:59:47.601817 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 13:59:47.628989 15426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240503 13:59:47.679702 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 13:59:47.764253 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 13:59:47.765622 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:47.806545 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 13:59:47.809635 15691 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 109 similar messages]
W20240503 13:59:47.823913 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 13:59:47.876245 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 13:59:47.888185 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 13:59:47.927847 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:59:47.928998 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 13:59:47.930195 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:47.933022 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:47.933261 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 13:59:47.947443 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 13:59:47.954582 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:47.968848 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:47.973882 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:47.987697 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 13:59:48.012156 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 13:59:48.039420 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 13:59:48.047324 13118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240503 13:59:48.048034 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 13:59:48.073227 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 13:59:48.106660 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 13:59:48.114499 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 13:59:48.149500 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 13:59:48.173103 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 13:59:48.197350 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 13:59:48.217877 13249 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 106 similar messages]
W20240503 13:59:48.218845 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 13:59:48.221745 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 13:59:48.248526 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 13:59:48.261557 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 13:59:48.266759 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 13:59:48.283166 12987 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240503 13:59:48.298177 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 13:59:48.320089 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 13:59:48.353530 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:48.355170 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:48.356230 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:48.366897 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:48.372115 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:48.378316 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:48.389457 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:59:48.402487 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:59:48.437139 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 13:59:48.447862 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 13:59:48.457537 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 13:59:48.459393 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 13:59:48.534400 15558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 93 similar messages]
W20240503 13:59:48.563802 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 13:59:48.601008 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 13:59:48.639137 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 13:59:48.681357 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 13:59:48.684706 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 13:59:48.695235 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 13:59:48.709977 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 13:59:48.748718 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240503 13:59:48.761754 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 13:59:48.766275 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:48.769927 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:48.774003 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:48.781968 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:59:48.781991 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:48.798209 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:48.805399 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:59:48.818315 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 13:59:48.839082 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 13:59:48.900811 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 13:59:48.904978 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 13:59:48.933002 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 13:59:48.954761 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:48.982650 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 13:59:49.001811 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 13:59:49.179556 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:49.180382 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:49.183362 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:49.186772 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:49.195616 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:49.205600 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:49.211931 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:49.216306 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 13:59:49.221875 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 13:59:49.299042 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 13:59:49.313571 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 13:59:49.504691 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:49.543859 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 13:59:49.548930 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 13:59:49.584784 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:49.588616 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:49.588927 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:49.600628 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:59:49.608191 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 13:59:49.614078 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:59:49.624079 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 13:59:49.624670 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:59:49.633026 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 13:59:49.645255 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 13:59:49.648638 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 13:59:49.649263 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:49.821532 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 13:59:49.923431 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 13:59:50.009982 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 13:59:50.014182 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 13:59:50.016829 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 13:59:50.020447 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:50.028620 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:50.029407 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:50.035826 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:59:50.036592 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:50.046281 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:59:50.065763 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 13:59:50.109004 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:50.138461 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 13:59:50.214267 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 13:59:50.252768 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 13:59:50.354429 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 13:59:50.361172 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 13:59:50.361232 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 13:59:50.367020 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 13:59:50.413805 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 13:59:50.418030 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:50.420666 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:50.426587 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:50.437316 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:50.438390 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:50.460662 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:59:50.462579 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 13:59:50.463303 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 13:59:50.465842 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 13:59:50.468600 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:50.487563 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 13:59:50.495865 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:50.523648 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 13:59:50.526216 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 13:59:50.543262 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 13:59:50.594352 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 13:59:50.605659 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 13:59:50.616703 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 13:59:50.631104 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 13:59:50.682893 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 13:59:50.697618 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 13:59:50.715852 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 13:59:50.732173 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 13:59:50.737377 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 13:59:50.748651 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 13:59:50.763931 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 13:59:50.764601 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 13:59:50.780625 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 13:59:50.854732 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:50.856393 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:50.859310 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:50.859496 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 13:59:50.869540 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 13:59:50.871470 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:50.888643 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:59:50.895052 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:50.900578 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 13:59:50.904677 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:50.910881 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 13:59:50.949544 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 13:59:51.036381 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 13:59:51.094604 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 13:59:51.133765 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 13:59:51.166190 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 13:59:51.194963 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 13:59:51.205291 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 13:59:51.267048 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:51.271734 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:51.274387 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:51.278734 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:51.285543 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:51.292274 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:59:51.292627 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 13:59:51.322923 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:51.340556 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 13:59:51.349695 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:51.349787 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 13:59:51.502911 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 13:59:51.524797 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 13:59:51.537288 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 13:59:51.653108 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 13:59:51.682652 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 13:59:51.690505 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:51.690742 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:51.701706 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:51.703924 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:51.707176 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:51.723814 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:51.727844 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:51.746832 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 13:59:51.774536 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 13:59:51.841660 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:51.846020 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 13:59:51.867312 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 13:59:51.917414 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 13:59:51.934594 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 13:59:52.035391 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 13:59:52.069247 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 13:59:52.098083 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:52.101464 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:52.103385 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:52.108474 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:52.109572 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:52.121085 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:52.126935 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:52.128245 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 13:59:52.169054 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 13:59:52.286195 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 13:59:52.315678 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 13:59:52.365684 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 13:59:52.399988 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 13:59:52.428922 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 13:59:52.431542 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:59:52.513113 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:52.514757 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:52.515197 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:52.520651 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:52.523016 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:52.533129 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:52.536806 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:52.562243 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 13:59:52.622256 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 13:59:52.650089 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 13:59:52.696995 15426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240503 13:59:52.719457 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 13:59:52.739825 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 13:59:52.787714 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 13:59:52.788784 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 13:59:52.810423 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 13:59:52.828421 15691 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 112 similar messages]
W20240503 13:59:52.833453 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:52.872351 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 13:59:52.873055 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 13:59:52.909598 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 13:59:52.931950 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:52.932066 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:52.936630 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:52.938117 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:52.941462 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:52.961197 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:52.977921 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:52.995232 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 13:59:52.999056 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 13:59:53.001847 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 13:59:53.010576 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 13:59:53.012537 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 13:59:53.031072 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 13:59:53.070349 13118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240503 13:59:53.135504 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 13:59:53.153573 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 13:59:53.187232 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 13:59:53.229082 13249 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 108 similar messages]
W20240503 13:59:53.229957 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 13:59:53.238438 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 13:59:53.242023 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 13:59:53.287786 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 13:59:53.288666 12987 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240503 13:59:53.328244 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 13:59:53.338176 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:53.340703 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240503 13:59:53.350114 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:53.355563 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:53.356935 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:53.361137 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:59:53.371706 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 13:59:53.374826 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:53.387652 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:53.388337 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:53.401093 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 13:59:53.477238 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 13:59:53.478190 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 13:59:53.501451 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 13:59:53.506034 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 13:59:53.533866 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 13:59:53.553758 15558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240503 13:59:53.571262 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 13:59:53.625638 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 13:59:53.637506 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 13:59:53.676090 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 13:59:53.738328 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 13:59:53.750440 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 68 similar messages]
I20240503 13:59:53.757288 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:53.759969 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:53.764346 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:53.781775 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:53.782650 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:53.786628 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:53.794222 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 13:59:53.804598 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 13:59:53.819131 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 13:59:53.819439 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:59:53.878610 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:53.897256 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 13:59:53.945482 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 13:59:53.946518 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 13:59:54.052901 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 13:59:54.088894 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 13:59:54.157398 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 13:59:54.187666 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:54.190663 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:54.196636 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:59:54.196828 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 13:59:54.207831 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:54.214612 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:54.222105 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:54.228828 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:54.296507 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 13:59:54.457024 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 13:59:54.470151 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:54.477954 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 13:59:54.501087 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240503 13:59:54.594437 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:54.600638 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:54.605304 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:59:54.611873 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240503 13:59:54.619009 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:59:54.619073 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:54.619657 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:59:54.629493 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240503 13:59:54.644644 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:59:54.652065 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 13:59:54.658298 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 13:59:54.673254 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 13:59:54.677316 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 13:59:54.740027 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 13:59:54.753381 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 13:59:54.819036 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 13:59:54.941318 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:59:55.000231 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:55.008368 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:55.010965 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:55.012187 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:55.018229 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:55.024720 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:55.025285 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:55.033010 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 13:59:55.049394 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 13:59:55.134285 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 13:59:55.142758 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 13:59:55.163462 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 13:59:55.177937 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 13:59:55.258714 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 13:59:55.275032 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 13:59:55.296018 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 13:59:55.334456 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 13:59:55.375672 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 13:59:55.406783 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:55.410257 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:55.415225 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:55.415935 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:55.427300 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:55.432729 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:59:55.452564 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:59:55.460981 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 13:59:55.477766 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 13:59:55.480450 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 13:59:55.489018 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 13:59:55.525470 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 13:59:55.527172 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:55.532575 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 13:59:55.583865 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 13:59:55.657845 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 13:59:55.674765 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 13:59:55.683207 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 13:59:55.718978 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 13:59:55.730952 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 13:59:55.756028 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 13:59:55.802130 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 13:59:55.805687 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240503 13:59:55.809487 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:55.810254 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:55.811671 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:55.820873 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:59:55.827661 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 13:59:55.829650 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:59:55.841853 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 13:59:55.843569 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:55.855607 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 13:59:55.872210 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 13:59:55.885658 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 13:59:55.923731 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 13:59:55.959537 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 13:59:55.966373 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:55.985610 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 13:59:56.048542 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 13:59:56.116899 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 13:59:56.131479 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 13:59:56.133818 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 13:59:56.152326 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 13:59:56.195205 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240503 13:59:56.212522 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:56.218307 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:56.224011 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:56.225091 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:56.232636 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:59:56.246368 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:56.256879 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:59:56.264178 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 13:59:56.279973 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 13:59:56.329365 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 13:59:56.391587 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:56.445957 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 13:59:56.618708 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 13:59:56.620265 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 13:59:56.631287 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:56.641656 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:56.641855 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:56.644223 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:56.651865 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:56.661720 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:56.664642 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:56.676465 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 13:59:56.759581 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 13:59:56.792063 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:56.837330 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 13:59:56.890117 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 13:59:56.911942 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 13:59:56.945624 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 13:59:57.018796 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 13:59:57.036417 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 13:59:57.040058 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 13:59:57.040810 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:59:57.046197 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 13:59:57.054575 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:57.056378 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:57.059851 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:57.066726 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:59:57.067262 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:59:57.076824 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 13:59:57.222736 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 13:59:57.255128 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 13:59:57.293730 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 13:59:57.297349 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 13:59:57.299389 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 13:59:57.305841 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 13:59:57.341281 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 13:59:57.393555 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 13:59:57.447089 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:57.449605 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:57.453444 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:57.456055 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:59:57.459450 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:57.463584 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:57.483477 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:59:57.590834 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 13:59:57.647375 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 13:59:57.674908 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 13:59:57.701787 15426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240503 13:59:57.702383 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 13:59:57.756392 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 13:59:57.760123 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 13:59:57.765278 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 13:59:57.766530 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 13:59:57.778725 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 13:59:57.832332 15691 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 110 similar messages]
W20240503 13:59:57.854082 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 13:59:57.866818 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 13:59:57.868541 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 13:59:57.873859 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:59:57.880538 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:57.881374 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 13:59:57.886711 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 13:59:57.887916 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 13:59:57.897790 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 13:59:57.897786 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 13:59:57.906877 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 13:59:57.908704 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 13:59:57.912117 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:59:58.000345 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:58.010792 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 13:59:58.049085 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 13:59:58.060519 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 13:59:58.071091 13118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240503 13:59:58.095495 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 13:59:58.128295 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 13:59:58.149473 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 13:59:58.189126 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 13:59:58.195449 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 13:59:58.212440 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 13:59:58.215087 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 13:59:58.218907 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 13:59:58.259310 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 13:59:58.277309 13249 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 111 similar messages]
W20240503 13:59:58.277966 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 13:59:58.287629 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:58.289109 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:58.289600 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:59:58.293145 12987 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 65 similar messages]
W20240503 13:59:58.295774 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 13:59:58.299166 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:58.300843 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:58.310425 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:58.323598 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:58.332808 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 13:59:58.351452 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 13:59:58.397711 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 13:59:58.417590 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 13:59:58.516280 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:58.526950 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 13:59:58.559391 15558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240503 13:59:58.560001 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 13:59:58.570170 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 13:59:58.580065 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 13:59:58.595569 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 13:59:58.666800 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 13:59:58.686359 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 13:59:58.691941 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:58.702060 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:58.707813 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:58.709818 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:58.710978 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:58.728124 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:58.733793 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:58.742249 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 13:59:58.796114 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 13:59:58.805500 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240503 13:59:58.821710 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 13:59:59.007105 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 13:59:59.024080 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 13:59:59.038524 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:59.086258 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 13:59:59.097404 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 13:59:59.109545 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:59.110649 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:59.111500 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:59.112573 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:59.122927 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:59.132709 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:59:59.132812 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 13:59:59.259081 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 13:59:59.326648 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 13:59:59.372437 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 13:59:59.501520 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 13:59:59.504951 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:59.508327 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:59.512583 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:59.513334 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:59.531315 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 13:59:59.535346 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:59.552361 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 13:59:59.567876 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 13:59:59.595259 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 13:59:59.608419 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 13:59:59.618099 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 13:59:59.621590 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 13:59:59.673300 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 13:59:59.690601 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 13:59:59.708325 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 13:59:59.716753 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 13:59:59.737980 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 13:59:59.780531 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 13:59:59.786504 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 13:59:59.887006 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 13:59:59.909878 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 13:59:59.919407 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 13:59:59.920807 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:59.921494 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 13:59:59.925887 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 13:59:59.936653 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 13:59:59.951010 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 13:59:59.992842 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 14:00:00.118410 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:00:00.162796 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 14:00:00.182015 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:00:00.186355 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:00:00.199330 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:00:00.202697 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:00:00.235281 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 14:00:00.274506 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:00:00.285084 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 14:00:00.293901 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:00:00.300233 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:00:00.309489 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 14:00:00.336669 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 14:00:00.352370 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 14:00:00.354643 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 14:00:00.365541 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 14:00:00.369993 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 14:00:00.370575 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 14:00:00.376236 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 14:00:00.380450 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:00:00.389740 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 14:00:00.399309 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 14:00:00.429661 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:00:00.455312 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:00:00.513923 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 14:00:00.652784 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 14:00:00.671188 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:00:00.690397 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 14:00:00.697851 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:00:00.724284 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:00:00.726552 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:00:00.737408 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:00:00.748207 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 14:00:00.757192 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 14:00:00.760645 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 14:00:00.762089 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:00.764094 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 14:00:00.764187 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:00.789088 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:00:00.809861 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 14:00:00.826118 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 14:00:00.826949 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:00:00.832180 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 14:00:00.840646 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 14:00:00.844018 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 14:00:00.888188 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:00:00.948521 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 14:00:00.963683 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 14:00:00.991457 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:00:01.008344 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 14:00:01.058912 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 14:00:01.079653 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 14:00:01.081331 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 14:00:01.180356 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 14:00:01.180452 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:01.180549 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 14:00:01.182415 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 14:00:01.185408 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:01.189867 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 14:00:01.192647 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 14:00:01.220746 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 14:00:01.240587 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:00:01.249938 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 14:00:01.386226 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 14:00:01.402855 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 14:00:01.461395 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:00:01.511283 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 14:00:01.558164 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:00:01.561687 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 14:00:01.578542 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:01.581580 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 14:00:01.584219 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 14:00:01.596994 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 14:00:01.604244 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 14:00:01.609735 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:00:01.610273 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 14:00:01.613703 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 14:00:01.667994 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 14:00:01.738797 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:00:01.871546 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:00:01.947074 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:00:01.958405 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 14:00:02.001741 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:02.009179 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:02.010664 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 14:00:02.015429 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 14:00:02.016963 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:00:02.021538 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 14:00:02.028456 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 14:00:02.032541 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:00:02.103565 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:00:02.207641 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:00:02.217495 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 14:00:02.260692 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:00:02.267313 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:00:02.287583 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:00:02.309056 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:00:02.326400 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 14:00:02.388026 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:00:02.409458 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 14:00:02.419567 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 14:00:02.420178 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 14:00:02.432801 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:02.433338 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:02.441602 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 14:00:02.443430 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 14:00:02.447373 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 14:00:02.502071 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:00:02.527513 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:00:02.581321 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:00:02.584537 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:00:02.624657 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:00:02.662436 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 14:00:02.704896 15426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 118 similar messages]
W20240503 14:00:02.728678 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:00:02.742899 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:00:02.749717 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:00:02.755771 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:00:02.780197 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:00:02.787626 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:00:02.801663 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:00:02.807997 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:00:02.810446 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240503 14:00:02.823352 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 14:00:02.825996 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:02.832083 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:02.833911 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:00:02.838071 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 14:00:02.850286 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 14:00:02.866487 15691 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 108 similar messages]
I20240503 14:00:02.866912 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 14:00:02.870281 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:00:02.976967 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:00:03.032680 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:00:03.047394 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:00:03.065565 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:00:03.100383 13118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240503 14:00:03.100988 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:00:03.194958 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:00:03.204833 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:00:03.205452 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 14:00:03.214951 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240503 14:00:03.245965 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 14:00:03.248848 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 14:00:03.249763 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 14:00:03.250273 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:03.251296 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 14:00:03.264276 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:03.272065 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 14:00:03.275522 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:00:03.277240 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:00:03.293203 12987 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240503 14:00:03.293869 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240503 14:00:03.294499 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 14:00:03.316460 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:00:03.319705 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:00:03.345624 12896 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240503 14:00:03.346309 12896 catalog_manager.cc:4573] TS e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2427783ee0e348b98453818704ece5f8: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 14:00:03.355814 13249 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 111 similar messages]
W20240503 14:00:03.383512 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:00:03.390712 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:00:03.415581 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:00:03.419673 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:00:03.539042 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 14:00:03.569830 15558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240503 14:00:03.572082 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:00:03.583541 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:00:03.590094 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 14:00:03.596997 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 14:00:03.650409 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 14:00:03.658087 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 14:00:03.658815 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240503 14:00:03.659855 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:03.672139 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:00:03.673650 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 14:00:03.673890 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 14:00:03.690326 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 14:00:03.705873 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 14:00:03.721112 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:00:03.742147 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:00:03.789407 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:00:03.847553 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240503 14:00:03.848131 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:00:03.904057 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:00:04.062610 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 14:00:04.079581 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:04.079607 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 14:00:04.088311 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 14:00:04.089388 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:04.102267 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 14:00:04.103153 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 14:00:04.108723 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240503 14:00:04.127128 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 14:00:04.137574 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 14:00:04.224603 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:00:04.295362 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:00:04.306165 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:00:04.413290 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:00:04.415882 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:00:04.429875 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240503 14:00:04.500511 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:04.501353 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 14:00:04.508045 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:00:04.519760 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 14:00:04.520867 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:04.523062 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 14:00:04.535524 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 14:00:04.537072 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 14:00:04.582600 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:00:04.607856 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:00:04.620529 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:00:04.668056 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:00:04.670219 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:00:04.674960 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:00:04.676545 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:00:04.687669 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:00:04.788512 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:00:04.811332 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:00:04.873185 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240503 14:00:04.904588 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:04.915704 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 14:00:04.915796 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:00:04.918160 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:04.919422 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 14:00:04.924196 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 14:00:04.924376 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 14:00:04.936570 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 14:00:04.953469 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:00:04.995983 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:00:05.163692 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:00:05.192956 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:00:05.201784 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:00:05.214968 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:00:05.221088 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:00:05.235520 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:00:05.275247 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240503 14:00:05.331581 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:05.336581 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 14:00:05.340312 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:00:05.340961 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 14:00:05.348768 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 14:00:05.358623 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 14:00:05.369380 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 14:00:05.385358 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 14:00:05.385824 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 14:00:05.398085 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:00:05.408970 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:00:05.424867 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:00:05.469859 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:00:05.473706 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:00:05.482496 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:00:05.489866 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:00:05.518070 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:00:05.629519 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:00:05.664613 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:00:05.701999 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:00:05.739939 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240503 14:00:05.741811 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 14:00:05.742486 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:05.742902 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 14:00:05.747399 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 14:00:05.749872 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:05.764062 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 14:00:05.770457 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:00:05.777513 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 14:00:05.777665 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 14:00:05.800153 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:00:05.826089 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:00:05.826354 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:00:05.840345 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:00:05.905483 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 14:00:05.925916 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:00:05.971033 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:00:05.985111 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:00:06.000833 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:00:06.007387 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:00:06.051443 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:00:06.070109 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:00:06.078184 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:00:06.104501 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:00:06.130103 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:00:06.130419 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 14:00:06.146416 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 14:00:06.148922 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 14:00:06.151618 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:06.163867 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 14:00:06.174775 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:00:06.181438 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:06.183113 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 14:00:06.269368 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:00:06.338218 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 14:00:06.347430 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:00:06.398878 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:00:06.501657 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 14:00:06.570999 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 14:00:06.571934 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:06.572291 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 14:00:06.573185 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:06.574532 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 14:00:06.576663 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 14:00:06.581264 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 14:00:06.603332 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 14:00:06.609526 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:00:06.719345 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:00:06.760771 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 14:00:06.775408 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:00:06.806136 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:00:06.923904 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:00:06.956072 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:00:06.956240 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 14:00:06.981256 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 14:00:06.988785 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 14:00:06.989152 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 14:00:07.003731 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 14:00:07.007359 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 14:00:07.009749 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:07.017999 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:00:07.018476 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 14:00:07.030913 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:00:07.043175 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:00:07.182828 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:00:07.208474 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:00:07.210865 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:00:07.229104 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:00:07.244453 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:00:07.321521 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:00:07.328701 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:00:07.342664 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:00:07.351505 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 14:00:07.374624 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 14:00:07.416504 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 14:00:07.416644 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 14:00:07.417088 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:00:07.419003 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 14:00:07.419279 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 14:00:07.419505 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 14:00:07.433640 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:00:07.435647 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240503 14:00:07.446193 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 14:00:07.517235 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:00:07.616358 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:00:07.627583 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:00:07.670009 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:00:07.670189 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:00:07.757246 15426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240503 14:00:07.771641 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:00:07.790310 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:00:07.795375 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:00:07.821753 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:00:07.829664 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 14:00:07.830369 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:07.833520 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 14:00:07.834381 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:07.835799 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 14:00:07.837509 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 14:00:07.842965 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 14:00:07.853983 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 14:00:07.873584 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 14:00:07.875651 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:00:07.905385 15691 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 111 similar messages]
W20240503 14:00:07.914137 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 14:00:07.951355 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:00:07.997917 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:00:08.049696 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:00:08.078999 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:00:08.092882 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:00:08.128105 13118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240503 14:00:08.131871 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:00:08.194836 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:00:08.205772 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:00:08.231289 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:00:08.234364 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:00:08.250157 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 14:00:08.260609 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:08.265138 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 14:00:08.270834 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:08.271548 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:00:08.281785 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 14:00:08.282019 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 14:00:08.304850 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 14:00:08.315253 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 14:00:08.356506 13249 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 113 similar messages]
W20240503 14:00:08.387534 12987 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 66 similar messages]
W20240503 14:00:08.388173 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:00:08.434043 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:00:08.465785 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:00:08.476912 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:00:08.493746 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 14:00:08.538688 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:00:08.551476 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:00:08.554081 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:00:08.580770 15558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240503 14:00:08.581476 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:00:08.594436 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:00:08.620754 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:00:08.635968 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240503 14:00:08.675197 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 14:00:08.677538 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:08.679677 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:08.682821 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:00:08.687054 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 14:00:08.695547 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 14:00:08.702838 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 14:00:08.711874 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:00:08.805382 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:00:08.806725 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:00:08.865607 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240503 14:00:08.908243 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:00:08.949143 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 14:00:08.960057 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 14:00:09.080654 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 14:00:09.081506 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:09.087702 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 14:00:09.093447 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 14:00:09.100754 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:09.105262 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:00:09.120893 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 14:00:09.126786 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:00:09.229552 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:00:09.265532 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:00:09.369302 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 14:00:09.384416 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:00:09.401129 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:00:09.412185 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:00:09.433975 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:00:09.434190 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240503 14:00:09.484908 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:09.485169 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 14:00:09.496731 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 14:00:09.499694 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 14:00:09.501709 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 14:00:09.502787 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:00:09.522718 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 14:00:09.523773 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 14:00:09.556207 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:00:09.602531 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:00:09.608543 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:00:09.679062 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:00:09.679852 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:00:09.711627 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:00:09.799399 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:00:09.828166 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 14:00:09.845328 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:00:09.856051 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240503 14:00:09.885874 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:09.891732 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:09.893534 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 14:00:09.896878 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:00:09.908653 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 14:00:09.916379 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 14:00:09.927592 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 14:00:09.932664 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 14:00:09.954667 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:00:10.020596 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:00:10.038328 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:00:10.056978 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:00:10.065402 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:00:10.070994 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:00:10.081660 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:00:10.137676 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:00:10.225631 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:00:10.246114 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:00:10.247998 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:00:10.250437 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:00:10.278369 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240503 14:00:10.326283 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 14:00:10.326891 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:10.330879 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 14:00:10.345132 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:00:10.350147 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 14:00:10.352118 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:10.352691 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 14:00:10.360966 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:00:10.361778 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 14:00:10.410286 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:00:10.428437 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 14:00:10.440896 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:00:10.486635 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:00:10.504096 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:00:10.638270 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:00:10.639465 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:00:10.661370 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:00:10.684206 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 14:00:10.727917 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:10.728792 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 14:00:10.732439 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240503 14:00:10.736661 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:00:10.737259 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:10.740757 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 14:00:10.756601 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 14:00:10.757521 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 14:00:10.768378 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 14:00:10.815977 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:00:10.855741 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 14:00:10.865419 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:00:10.874333 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:00:10.875845 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:00:10.900797 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:00:10.939245 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:00:10.984809 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:00:10.993607 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:00:11.034701 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:00:11.086230 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 14:00:11.134090 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 14:00:11.134861 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 14:00:11.137079 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:00:11.137528 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:11.138868 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 14:00:11.138929 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 14:00:11.161581 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:00:11.163502 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 14:00:11.166810 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 14:00:11.202957 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:00:11.276943 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:00:11.305133 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:00:11.311949 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:00:11.394477 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:00:11.409070 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 14:00:11.567858 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 14:00:11.569208 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 14:00:11.570348 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 14:00:11.586261 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:11.588733 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:11.601356 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 14:00:11.612131 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 14:00:11.626163 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:00:11.678444 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:00:11.770725 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:00:11.816345 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:00:11.838701 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:00:11.839740 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:00:11.886992 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:00:11.900547 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:00:11.922956 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:00:11.971338 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 14:00:11.977737 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:11.987406 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 14:00:11.990782 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 14:00:11.991843 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
W20240503 14:00:12.005932 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 14:00:12.014210 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:00:12.015432 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:12.015764 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 14:00:12.024513 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 14:00:12.085573 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:00:12.104863 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:00:12.109911 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:00:12.136401 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:00:12.156006 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:00:12.222254 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:00:12.316627 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:00:12.336477 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:00:12.352548 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 14:00:12.396796 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 14:00:12.399829 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 14:00:12.402547 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:00:12.403919 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 14:00:12.410153 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 14:00:12.417080 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 14:00:12.417413 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 14:00:12.418548 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 14:00:12.429319 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 14:00:12.438663 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 14:00:12.442399 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:00:12.443194 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:00:12.470003 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:00:12.471518 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:00:12.491544 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:00:12.512238 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:00:12.547232 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:00:12.563192 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:00:12.651422 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:00:12.755223 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 14:00:12.813316 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:12.813544 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 14:00:12.814422 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:12.821326 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 14:00:12.828783 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:00:12.829034 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 14:00:12.833190 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 14:00:12.856559 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 14:00:12.858275 15426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 122 similar messages]
W20240503 14:00:12.873171 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:00:12.944617 15691 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 111 similar messages]
W20240503 14:00:12.945444 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:00:12.993500 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:00:12.993767 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:00:13.061332 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:00:13.066618 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:00:13.082299 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:00:13.088516 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:00:13.118410 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:00:13.144744 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:00:13.190166 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:00:13.192324 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:00:13.202502 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 14:00:13.214030 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:13.220393 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 14:00:13.220621 13118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240503 14:00:13.221711 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:13.223248 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 14:00:13.226481 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 14:00:13.231320 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 14:00:13.252331 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 14:00:13.277030 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 14:00:13.289235 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 14:00:13.301290 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:00:13.326462 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:00:13.399116 12987 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240503 14:00:13.413203 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 14:00:13.437979 13249 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 108 similar messages]
W20240503 14:00:13.439503 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:00:13.439571 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:00:13.495568 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:00:13.555888 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:00:13.583207 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:00:13.603945 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:00:13.607554 15558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240503 14:00:13.608953 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 14:00:13.643162 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 14:00:13.643427 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 14:00:13.652571 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 14:00:13.652944 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 14:00:13.662662 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 14:00:13.665057 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:13.673261 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:13.693701 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 14:00:13.717913 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:00:13.849529 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:00:13.885502 13380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 73 similar messages]
W20240503 14:00:13.970275 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:00:14.009500 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:00:14.036516 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 14:00:14.037379 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 14:00:14.059216 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 14:00:14.060637 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 14:00:14.063098 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 14:00:14.064544 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240503 14:00:14.065233 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:14.068661 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:14.100564 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 14:00:14.104558 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 14:00:14.156808 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:00:14.211282 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:00:14.248137 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:00:14.258913 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:00:14.259742 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:00:14.282248 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:00:14.404129 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:00:14.407114 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:00:14.424959 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 14:00:14.435768 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:00:14.455300 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:00:14.471282 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:00:14.483940 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:00:14.485538 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 14:00:14.485713 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 14:00:14.486783 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 14:00:14.492321 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 14:00:14.500212 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:14.513649 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:00:14.515672 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 14:00:14.516757 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 14:00:14.544189 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:00:14.610821 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:00:14.632233 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:00:14.633970 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:00:14.781642 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:00:14.868428 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:00:14.879478 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 14:00:14.907953 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 14:00:14.913049 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 14:00:14.920301 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 14:00:14.924883 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:00:14.925181 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 14:00:14.925465 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 14:00:14.940827 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 14:00:14.940977 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 14:00:14.956051 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 14:00:15.012063 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:00:15.026674 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:00:15.028875 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:00:15.066726 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:00:15.110906 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:00:15.125149 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:00:15.155721 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:00:15.259975 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:00:15.262416 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:00:15.262576 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:00:15.285153 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 14:00:15.330987 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 14:00:15.335489 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 14:00:15.336362 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:00:15.340453 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 14:00:15.344187 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:15.353102 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:15.353574 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 14:00:15.414430 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:00:15.421628 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 14:00:15.479964 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:00:15.488806 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:00:15.507082 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:00:15.550485 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:00:15.558116 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:00:15.587705 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:00:15.606072 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:00:15.608253 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:00:15.645602 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:00:15.680536 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:00:15.726845 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 14:00:15.733072 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:15.739624 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 14:00:15.743526 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:00:15.744742 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 14:00:15.750228 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 14:00:15.765349 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 14:00:15.773739 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 14:00:15.812435 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:00:15.883697 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:00:15.888003 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:00:15.893324 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:00:15.897467 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:00:15.937877 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:00:15.959470 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 14:00:15.961154 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:00:15.984516 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:00:16.014678 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:00:16.018397 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:00:16.128297 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 14:00:16.159092 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:16.162297 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:00:16.163352 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 14:00:16.167634 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 14:00:16.173388 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:16.174405 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 14:00:16.178213 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 14:00:16.190739 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 14:00:16.309749 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:00:16.322999 15426 consensus_peers.cc:485] T 3be039d9c2294eb5bec575fc0505280c P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:00:16.351336 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:00:16.389726 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 14:00:16.422207 13249 consensus_peers.cc:485] T 2b362fed86c042868fc5b2fbaf83b369 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:00:16.433310 15558 consensus_peers.cc:485] T dd39f3ab7cdb42b38267214056605328 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:00:16.489526 13249 consensus_peers.cc:485] T 866723d26350414197ec27544277db9c P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:00:16.525799 15691 consensus_peers.cc:485] T ab1fed738e1c46a6a0a0f9e2b9cfd9b0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 14:00:16.574559 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 14:00:16.578248 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 14:00:16.581403 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 14:00:16.587520 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 14:00:16.615847 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:16.620380 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:16.620911 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 14:00:16.650650 15691 consensus_peers.cc:485] T bff36bb937a149c2890467aed1e54bca P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 14:00:16.670423 13118 consensus_peers.cc:485] T cbd6d38ea18b484d80f4a918148efd3a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:00:16.698477 13380 consensus_peers.cc:485] T 4b4b26adad184e1ea57c00a44635679c P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:00:16.740393 15558 consensus_peers.cc:485] T 2095ba0a3e264d79bdc7f3dd3fc6a3b5 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 14:00:16.775662 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 14:00:16.803985 13249 consensus_peers.cc:485] T 477c33f203f649019eff2b28524fad90 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 14:00:16.874890 13118 consensus_peers.cc:485] T a6a94e83461643d4adfc4abc7dfb74e7 P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:00:16.948376 15691 consensus_peers.cc:485] T 8e06d93b10e34b4aa98899c2e9d6e16f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:00:16.955741 15426 consensus_peers.cc:485] T eb92a057165f4da7b96d910d847ff761 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:00:16.983402 15691 consensus_peers.cc:485] T 0f9224c2e3b946d993a5fa5c95a1d681 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:00:16.996214 13380 consensus_peers.cc:485] T 6f20d5faec5940ca8b7cffa124e51e4d P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240503 14:00:17.002418 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 14:00:17.010161 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:17.014125 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 14:00:17.015053 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 14:00:17.018481 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:17.020120 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 14:00:17.028533 13249 consensus_peers.cc:485] T 84f1ab11e74045f8a9d7033c5ead0c14 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 14:00:17.029592 15691 consensus_peers.cc:485] T a2b9dcff1b72490397f2afc5348923c1 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 14:00:17.036063 12987 consensus_peers.cc:485] T 38596c626067458495846931e0ecb7eb P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 14:00:17.050971 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 14:00:17.081266 15426 consensus_peers.cc:485] T db5568722b6344d18d6d6d271f05fc7a P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:00:17.102742 13249 consensus_peers.cc:485] T a746bcb2d718433f8c9acc8bd027a429 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:00:17.163767 15558 consensus_peers.cc:485] T 9dd22ebd569c4d1da07bcb3365973406 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:00:17.241364 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 14:00:17.290796 12987 consensus_peers.cc:485] T 48f1801af37a472d9b0479122cb1d27e P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:00:17.309242 15691 consensus_peers.cc:485] T a9a1258c39a144149bf5d03e9950dc06 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:00:17.310972 12987 consensus_peers.cc:485] T 7b57f64985b74e5fa75a5603ab8cc4bf P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 14:00:17.333878 15426 consensus_peers.cc:485] T 7519f1091c6e4bdda68e8c7bafcff9f5 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:00:17.347160 13380 consensus_peers.cc:485] T a78c6abae54349e8b3de6def2d7aab5f P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 14:00:17.383347 15426 consensus_peers.cc:485] T 25d41dbdacc341ee93c86f7a7f056123 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:00:17.385027 13118 consensus_peers.cc:485] T e020e9ba80c34c2d90950f6570d1a0ab P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240503 14:00:17.408903 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 14:00:17.420701 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 14:00:17.422709 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 14:00:17.423337 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:00:17.427377 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:17.437727 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:17.461282 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 14:00:17.499343 15691 consensus_peers.cc:485] T 76f1450a07f34baaa8ecab16340e0ee0 P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 14:00:17.524309 13249 consensus_peers.cc:485] T bb4f0bb47a29437090750c4486df2f02 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:00:17.550091 15426 consensus_peers.cc:485] T 289d31dd1f8f40d9b87d7ad401d2d8dd P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 14:00:17.603693 15558 consensus_peers.cc:485] T 8438207c32194339a43e203f751d2345 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 14:00:17.651743 15691 consensus_peers.cc:485] T f21a51b44e0c40389e68f9e88c427bae P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 14:00:17.672354 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 14:00:17.697531 13249 consensus_peers.cc:485] T 397f0e3e877d4acd9157bdf99bee6255 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:00:17.737529 13249 consensus_peers.cc:485] T 05ad17680502464ba4b195fa8b0c2dfc P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 14:00:17.778959 15691 consensus_peers.cc:485] T bc1a69158cd14828864659dd55ae9aff P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 14:00:17.796052 15691 consensus_peers.cc:485] T 604875eff3594876af54e9126057bd7f P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 14:00:17.839556 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 14:00:17.843791 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:17.850906 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 14:00:17.851926 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:00:17.856928 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:17.860479 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 14:00:17.879298 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 14:00:17.890575 15426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240503 14:00:17.917227 15426 consensus_peers.cc:485] T 27d5ced54d2e4aa9b6ea01fe8700443d P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 14:00:17.929677 15426 consensus_peers.cc:485] T bf4ea1072ad542d89fda48ca4e839c79 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 14:00:17.964234 13249 consensus_peers.cc:485] T aa97788f7cdb4caaa243a609b34f55cf P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:00:17.977237 15691 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 113 similar messages]
W20240503 14:00:17.986939 12987 consensus_peers.cc:485] T 2f2c47eed62b4bec89714eb6f293ce7b P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:00:18.053362 15691 consensus_peers.cc:485] T 3da6bc0d649643e1acd8d8811ad1f5ad P 76c386bef61a4bd5a907266a5118ffe2 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:00:18.091320 15426 consensus_peers.cc:485] T 9dbcb7be2adb4655829c190c6cc6af85 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 14:00:18.152988 13118 consensus_peers.cc:485] T ff21b7dd34b54f5f92a8300e81bc5b1c P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:00:18.154556 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
W20240503 14:00:18.174448 15558 consensus_peers.cc:485] T 3a140864f59b483bb5fea3de83d28ecf P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:00:18.196269 12987 consensus_peers.cc:485] T 7dca80c8a74a42c4b459315878ef5d14 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:00:18.224748 13118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 62 similar messages]
I20240503 14:00:18.256825 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240503 14:00:18.266923 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 14:00:18.267146 13380 consensus_peers.cc:485] T 324217b2f526432a85caa80935941130 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240503 14:00:18.270843 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:18.274410 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:00:18.275851 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 14:00:18.277025 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 14:00:18.296317 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 14:00:18.311503 15426 consensus_peers.cc:485] T c7d5e8c09ec14ac7891c8a63cb896878 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 14:00:18.365808 13249 consensus_peers.cc:485] T d50d05d432e84d5eb0e54d1a68cdf158 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:00:18.369046 15558 consensus_peers.cc:485] T 10ee7dee14364389871900670a02906a P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 14:00:18.390839 13118 consensus_peers.cc:485] T d3849e4effe64656bd8b0f504c9dc17a P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:00:18.398758 15426 consensus_peers.cc:485] T c2df7186d52a46fea36c672a22a1a0f0 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:00:18.399005 13380 consensus_peers.cc:485] T 5edef9d909ac42d6b5acc22464a284c8 P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:00:18.417451 13249 consensus_peers.cc:485] T a5a875bb0dce46f5883c302ea1729430 P ba0194098aa84449ae17e8a0d4c43703 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 14:00:18.483587 13249 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 110 similar messages]
W20240503 14:00:18.495184 15558 consensus_peers.cc:485] T ce95d734e66b45eaba52b660a456f16e P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 14:00:18.586560 13118 consensus_peers.cc:485] T 3f959d7bf8b64d8d8737a2223b9798ea P 8ccf5a4ac450414c9d3579bd5800b770 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:00:18.592927 12987 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240503 14:00:18.594607 12987 consensus_peers.cc:485] T f58fc3bc579748bb88de34612808ccda P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 14:00:18.629374 12987 consensus_peers.cc:485] T 9bef98a0c5e0411cb55141a93c62f5c0 P 60ba3ffe3b7746469834cd91c84be14e -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:00:18.636191 15426 consensus_peers.cc:485] T 3e30d88e2c054de6b87e4beb5eb14019 P 61b887f0d40d497badeca5604c1f0546 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 14:00:18.640985 15558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240503 14:00:18.642239 15558 consensus_peers.cc:485] T 7bdb9c14caf04abca39e56f8bcf1d1d4 P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 14:00:18.679749 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 14:00:18.680042 15604 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240503 14:00:18.680599 15738 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:18.683290 13034 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:00:18.686427 15473 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
W20240503 14:00:18.704528 13380 consensus_peers.cc:462] T c047c61f42f04484842125847ca677aa P 7690ee79577c45e3a2c8f8b552c0bff8 -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111)
I20240503 14:00:18.704669 13295 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 14:00:18.709038 15558 consensus_peers.cc:485] T 557f14aaf5564ebaa461720a4d7997ff P 009f849ee7b54c80a3f5264695047e4d -> Peer e47105a0f46e4511b7112e571688a4c7 (127.6.98.72:44753): Couldn't send request to peer e47105a0f46e4511b7112e571688a4c7. Status: Network error: Client connection negotiation failed: client connection to 127.6.98.72:44753: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 14:00:18.709630 13164 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 14:00:18.736861 13380 consensus_peers.cc:485] T 5f34ac73ebe544a1b7a34e8c2d11568e P 7690ee79577c45e3a2c8f8b552c0bff8 ->